Re: [RFC net] tls: TLS_SW sendfile() stalls at large MSS

[Jiayuan Chen]

On 6/4/26 1:19 AM, Mike Fara wrote:
> Hi,
>
> Software-kTLS (TLS_SW) TX over sendfile()/splice() drops to the TCP
> persist-timer cadence (tens of KB/s, with individual sendfile() calls blocking
> for tens of seconds) when the path MSS is large -- e.g. loopback (MSS 65483) or
> jumbo frames. At a typical 1448-byte MSS it does not occur. Plain TCP
> sendfile() on the same path is unaffected, and kTLS write() (no splice) is
> unaffected, so it is specific to TLS_SW + the splice/sendfile path.
>
> It triggers only on large-MSS paths with software kTLS (no NIC TLS offload), so
> it is a niche path -- but it is a clean, reproducible multi-order-of-magnitude
> cliff, so it seems worth a look. Reproduces on current mainline. CCing David
> Howells as the author of the 2023 sendpage->MSG_SPLICE_PAGES splice_to_socket()
> rework referenced below, and Eric/Paolo as this is as much a TCP-corking
> interaction as a TLS one.
>
> Environment
> -----------
> - net/tls TLS_SW (no NIC offload; ethtool tls-hw-tx-offload: off [fixed]).
> - AES-GCM; gcm(aes) resolves to generic-gcm-vaes-avx512.
>
> Reproducer (no OpenSSL/handshake; TLS_TX programmed with a fixed key, the
> receiver discards ciphertext, like tools/testing/selftests/net/tls.c):
>
>      cc -O2 -Wall -o ktls_sendfile_stall ktls_sendfile_stall.c
>      ./ktls_sendfile_stall          # default loopback MSS (65483)
>      ./ktls_sendfile_stall 1448     # clamp sender MSS via TCP_MAXSEG
>
> Observed (loopback, single box):
>
>      MSS=default    sent=     4.0 MiB in  52.08s  =>  0.0001 GiB/s   (stalled)
>      MSS=1448       sent=  2048.0 MiB in   1.65s  =>  1.2106 GiB/s
>
> i.e. ~four orders of magnitude; at the default MSS a single sendfile() blocks
> for tens of seconds. For contrast, on the same loopback path:
>
>      plain TCP sendfile() (no TLS ULP):           7.87  GiB/s
>      kTLS write() (TLS_SW, no splice, 2 GiB):      1.99  GiB/s


I tested your ktls_sendfile_stall.c under stable 6.6 and upstream, but 
both of them works correctly.

~/code/tmp$ ./ktls_sendfile_stall 1448 MSS=1448 sent= 2048.0 MiB in 
2.32s => 0.8603 GiB/s ~/code/tmp$ ./ktls_sendfile_stall 1448 MSS=1448 
sent= 2048.0 MiB in 2.37s => 0.8439 GiB/s ~/code/tmp$ 
./ktls_sendfile_stall MSS=default sent= 2048.0 MiB in 1.64s => 1.2204 
GiB/s :~/code/tmp$ ./ktls_sendfile_stall MSS=default sent= 2048.0 MiB in 
1.70s => 1.1737 GiB/s ~/code/tmp$ ./ktls_sendfile_stall 1448 MSS=1448 
sent= 2048.0 MiB in 2.33s => 0.8570 GiB/s

~/code/tmp$ ethtool -k lo | grep "tls-hw-tx-offload" tls-hw-tx-offload: 
off [fixed]