[PATCH RFC 0/3] RDMA: Post small ULP command messages inline

[Andrei-Alexandru Bleortu]

The iSER initiator, the SRP initiator and the RTRS client each send a
small per-command message by reference, on a QP that asks for no
inline data. The HCA therefore has to DMA-read the message from host
memory before it can put the command on the wire.

Each patch asks for a little inline data when creating the QP and
creates it without inline data if the device refuses. SRP asks for
116 bytes and tries smaller sizes before none. The message goes out with
IB_SEND_INLINE when it fits, with the SGE pointing at the buffer's
kernel virtual address. The messages are:

  iSER  PDUs without a data segment (76 bytes of headers)
  SRP   SRP_CMD IUs without immediate data
  RTRS  read requests, about 60 bytes for RNBD

I measured them on two Xeon Platinum 8360Y hosts with ConnectX-6 Dx
(RoCEv2, 100 Gb/s), with the patches backported to an Ubuntu 6.8 kernel
(same added and removed lines). Every QP was granted 120 bytes of inline
data. Inlining lowered fio's mean 4 KiB random-read latency at queue
depth 1 (io_uring, null_blk target, fio's lat, which includes
submission) by:

  iSER, PDU posted alone              0.479 us (95% CI 0.270-0.689)
  iSER, behind a registration WR      0.701 us (0.375-1.026)
  RNBD over RTRS                      0.348 us (0.309-0.387)
  SRP                                 0.383 us (0.320-0.446)

Each is the mean of 15 paired runs, 10 for SRP. The second iSER line is
with always_register=Y. Each patch inlines the message even when
registration WRs go ahead of it in the same call, since it still saved
time there. Tell me if you would rather inline only when it is posted
alone. Before going further I would like to know whether you want these.

I am sending the same change for the NVMe/RDMA host's command capsule to
linux-nvme. An earlier version of it lowered fio's mean 4 KiB
random-read latency at queue depth 1 by 0.998 us (95% CI 0.943-1.049)
between two AMD EPYC 7452 hosts with ConnectX-5 Ex. Between two Xeon
Platinum 8360Y hosts with ConnectX-6 Dx it fell by 0.589 us
(0.558-0.618). Both used RoCEv2 at 100 Gb/s. I could not find earlier
discussion of inline sends for these ULPs on linux-rdma.

All three build with W=1 on rdma for-next without new warnings, and
the backports ran on the hosts above. RTRS also ran in a QEMU guest
with rdma_rxe, where an RNBD client and server on one host mapped a
null_blk device. 8 MiB of 4 KiB O_DIRECT writes and reads compared
equal, and 2050 of the 2051 read requests went inline. With rxe changed
to refuse inline data, QP creation fell back, every request went by
reference and the data still compared equal.

I wrote these patches with an LLM coding assistant (Claude Code),
which also drafted the commit messages and ran the build and the VM
test.

Andrei-Alexandru Bleortu (3):
  IB/iser: Post header-only PDUs inline
  RDMA/srp: Post command IUs inline
  RDMA/rtrs: Post client read requests inline

 drivers/infiniband/ulp/iser/iscsi_iser.h |   2 +
 drivers/infiniband/ulp/iser/iser_verbs.c |  29 ++++++-
 drivers/infiniband/ulp/rtrs/rtrs-clt.c   |  17 +++-
 drivers/infiniband/ulp/rtrs/rtrs-pri.h   |  10 ++-
 drivers/infiniband/ulp/rtrs/rtrs-srv.c   |   2 +-
 drivers/infiniband/ulp/rtrs/rtrs.c       |  22 +++--
 drivers/infiniband/ulp/srp/ib_srp.c      | 102 ++++++++++++++++++++---
 drivers/infiniband/ulp/srp/ib_srp.h      |   9 ++
 8 files changed, 170 insertions(+), 23 deletions(-)


base-commit: 978361dd2ccff357d4fe6aac6ce057d1bfae8312
-- 
2.56.0