Re: mlx5e:zero-prefix corruption during TCP DMA-BUF RX - help requested

[jiabin deng]

Hi Dragos,

Thank you for pointing us to this fix.

Our original test used Ubuntu kernel 7.0.0-30-generic, based on
v7.0.12, and did not include that change. We have now repeated the
test using Ubuntu's mainline build of Linux 7.2.6
(7.2.6-070206-generic), with its in-tree mlx5_core driver.

I checked the v7.2.6 source: it contains the small-frame guard from
commit e2466392a0b8496000e12181cb1ee1535eb0da25  In
mlx5e_handle_rx_cqe_mpwrq_shampo(), frames with
cqe_bcnt <= ETH_ZLEN + 2 * VLAN_HLEN set match = false and
flush = true [2].

The zero-filled corruption ending at the next 64-byte boundary in
GPU memory still reproduces with this kernel.

We kept the same sender and receiver data-checking logic, with
tcp-data-split on and rx-gro-hw on. One additional environment
change is that the NVIDIA GPU driver was updated from 595.71.05 to
595.91.07. The NIC firmware remains 32.43.2400 (MT_0000001117).

We repeated the same eight message sizes, from 32 KiB to 4 MiB,
with 50 iterations per size:

- All 400 iterations completed.
- 105 iterations passed; 295 contained corrupted data.
- 21,599 corrupted ranges were recorded, all zero-filled and
  ending at the next 64-byte boundary in GPU memory.
- For each size from 256 KiB through 4 MiB, all 50 iterations
  contained corruption.
- The full-message snapshots taken before and after
  cuFlushGPUDirectRDMAWrites() were identical in all iterations.

For example, in the first 32 KiB iteration, an 8-byte corrupted
range starts at offset 2147495416 bytes from the beginning of the
GPU buffer. This offset is 56 bytes into a 64-byte-aligned block,
so the eight zero bytes end exactly at the next 64-byte boundary.

These results come from the reproducer's runtime byte comparisons.

Could you suggest the next targeted checks or additional debug
information that would help narrow this down?

Thanks,
Jiabin Deng


On Tue, Sep 15, 2026 at 8:21 PM Dragos Tatulea <dtatulea@xxxxxxxxxx> wrote:
>
> Hi Jiabin,
>
> On 15.09.26 03:49, jiabin deng wrote:
> > Hello mlx5e and networking maintainers,
> >
> > We are investigating reproducible data corruption: with SHAMPO enabled,
> > some TCP data is corrupted when received through mlx5e directly into
> > CUDA-exported GPU DMA-BUF memory.
> >
> > The full expected number of bytes is received, but the prefixes of some
> > payloads become zero. The corrupted ranges end exactly at the next
> > 64-byte boundary in GPU memory.
> >> [...]
> >
> > Each iteration uses a new TCP connection. Test connections do not run
> > concurrently.
> >
> > Observed results:
> >
> > We tested eight message sizes from 32 KiB to 4 MiB, with 50 iterations
> > per size:
> >
> > - 400 iterations completed.
> > - 354 iterations contained at least one corrupted range.
> > - 20,707 corrupted ranges were recorded.
> > - All recorded corrupted ranges were zero-filled prefixes ending
> >   at the next 64-byte boundary in GPU memory.
> > - Corruption was present in both snapshots, before and after the
> >   explicit GPUDirect write flush.
> >
> > A specific example:
> >
> > - Packet payload length: 1448 bytes.
> > - Offset of the first payload byte from the start of the GPU buffer:
> >   2149169912 bytes.
> > - This byte offset modulo 64 bytes is 56 bytes.
> > - First 8 bytes: zero.
> > - Remaining 1440 bytes: correct.
> >
> This sounds a bit like the ethernet padding issue that was recently
> fixed [1]. Did you have this patch when testing?
>
> >
> > [...]
> >
>
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit?id=e2466392a0b8496000e12181cb1ee1535eb0da25
>
> Thanks,
> Dragos