Re: [PATCH RFC 6/8] SUNRPC: Reduce rpciod workqueue contention

[Chuck Lever]

On Mon, 14 Sep 2026, Tim Menninger wrote:
> If I am understanding correctly, the SLUB/barn/sheaf behavior we are
> looking at here is pre-existing rather than introduced by this series.
> The SMT affinity change is what changes the placement enough to expose
> the remote-free cost on this workload.

That is the basic idea, but there are nuances. Split it into
three pieces with different owners:

1. The list_lock contention is pre-existing and does not bound
throughput. Your numbers show it directly:

                            unpinned    node0 pinned
    contentions             3.346M         3.353M
    aggregate wait          5.05 min       4.01 min
    node 0 idle             0-2%           66%
    throughput              43.4 GB/s      46.5 GB/s

The lock is taken just as often in the run that has full
throughput and two thirds of node 0 idle.

I have since reproduced the acquisition rate here, on a single-node
EPYC 8224P (24c/48t) reading direct 1M blocks over NFS/RDMA at
12 GB/s, with the barn depth made a runtime knob and
CONFIG_SLUB_STATS on. At depth 1, 28% of barn puts fail and
free_add_partial takes list_lock 5.0M times per 10 seconds, the
same order as your 3.35M contended acquisitions. Total wait on the
lock in that window was 36 ms, and throughput did not change at any
depth from 1 to 64, nor with the work spread across all 48 CPUs,
nor with 16x the in-flight population at iodepth 256.

So the acquisition rate is ordinary. What is not ordinary is your
72 us average wait, against about 7 ns here. That is an allocator
question, and I'd like to take it to linux-mm with your numbers.
More on that below.

2. The throughput loss in the all-local placement comes from the
70M remote frees per 10 seconds, not from the lock. Pinning the
workload to node 0 removed the remote frees, returned 3 GB/s, and
left node 0 two thirds idle with the lock profile unchanged. That
cost is a function of where the CQs land relative to the
application threads, which the RDMA core decides at connect time,
and it exists with or without this series.

3. The smt scope costs a further 4 GB/s in that same placement
(43 vs 47 on your scope ladder), and nothing in the slab data
explains it: the frees stay on node 0 under both smt and
cache_shard, so the scope is not what moves them across nodes. I
don't have an explanation.

Since you have a robust reproducer, can you take this to the
linux-mm list (cc me) with the following three items (no new runs
needed for the first two):

- The debug config. Any of these lengthens every list_lock hold
  without disabling sheaves:

      zcat /proc/config.gz | grep -E \
          'DEBUG_SPINLOCK|DEBUG_LIST|PROVE_LOCKING|LOCK_STAT|KASAN'

- The CPU model and socket count.

- A cmpxchg_double_fail delta from /sys/kernel/slab/nfs_page over
  the same 10-second window as your other counters. __slab_free
  drops list_lock and retries when the freelist cmpxchg fails, so
  a large number there means each logical free costs several lock
  acquisitions. Mine was under a hundred.

One more thing I can't reconcile: 4.01 minutes of wait inside a
10-second window is 240 CPU-seconds, half of node 0's 48 threads
for the whole window, which does not fit 66% idle. Were the perf
lock capture and the mpstat sample the same 10 seconds, and did
perf lock cover all 96 CPUs or only node 0? If the wait is real,
node 0 spends half its time spinning even in the good run, and
the lock is where the headroom goes once the remote frees load
the node. If it is an artifact of the window, the lock costs less
than the profile says, and the report to linux-mm is a simpler
one.


-- 
Chuck Lever (Come to NFS bake-a-thon! https://nfsv4bat.org)