RE: [RFC PATCH 0/3] mm/vmscan: adaptive multi-threaded kswapd for NUMA-aware reclaim

[Ababneh, Ehab]

> -----Original Message-----
> From: Zi Yan <ziy@xxxxxxxxxx>
> Sent: Tuesday, September 8, 2026 7:16 PM
> To: Ababneh, Ehab <ehab.ababneh@xxxxxxxxx>; linux-mm@xxxxxxxxx; linux-
> kernel@xxxxxxxxxxxxxxx
> Cc: Buddy Lumpkin <buddy.lumpkin@xxxxxxxxxx>; Andrew Morton
> <akpm@xxxxxxxxxxxxxxxxxxxx>; Kairui Song <kasong@xxxxxxxxxxx>; Qi Zheng
> <qi.zheng@xxxxxxxxx>; Shakeel Butt <shakeel.butt@xxxxxxxxx>; Barry Song
> <baohua@xxxxxxxxxx>; Axel Rasmussen <axelrasmussen@xxxxxxxxxx>;
> Yuanchu Xie <yuanchu@xxxxxxxxxx>; Wei Xu <weixugc@xxxxxxxxxx>; David
> Hildenbrand <david@xxxxxxxxxx>; Lorenzo Stoakes <ljs@xxxxxxxxxx>; Liam R.
> Howlett <liam@xxxxxxxxxxxxx>; Vlastimil Babka <vbabka@xxxxxxxxxx>; Mike
> Rapoport <rppt@xxxxxxxxxx>; Suren Baghdasaryan <surenb@xxxxxxxxxx>;
> Michal Hocko <mhocko@xxxxxxxx>; Brendan Jackman
> <jackmanb@xxxxxxxxxx>; Johannes Weiner <hannes@xxxxxxxxxxx>
> Subject: Re: [RFC PATCH 0/3] mm/vmscan: adaptive multi-threaded kswapd
> for NUMA-aware reclaim
> 
> On Tue Sep 8, 2026 at 6:10 PM EDT, Ehab Ababneh wrote:
> > This series revives Buddy Lumpkin's earlier multi-kswapd proposal:
> > https://lkml.iu.edu/hypermail/linux/kernel/1804.0/00342.html
> +others
> 
> Please Cc everyone in the cover letter, so people can know the motivation of
> the patchset.
> 
> >
> > The motivation is stronger now than when the patch was first discussed.
> > Many current systems have hundreds of cores per NUMA node, not the
> > single-digit or low-tens core counts that were more common at the time.
> > When reclaim does not keep up, direct reclaim can still push
> > allocation latency into application paths and leave substantial CPU
> > capacity waiting for memory to be freed.
> >
> > This patchset adds adaptive multi-threaded kswapd. The wakeup policy
> > uses node load to decide how many kswapd workers to run, so reclaim
> > can scale when it helps and stay conservative on already busy nodes.
> >
> > Series summary:
> >
> > 1. Allow multiple kswapd threads per node and add control plumbing.
> > 2. Wake an appropriate number of kswapd threads from per-node
> >    runnable load.
> >
> > Concerns from the original discussion and how this series addresses
> > some of them:
> >
> > - Concern: Direct reclaim is intended to slow a memory-hogging thread.
> >   Response: That can be acceptable on lower-core systems. On high-core
> >   systems, idling many cores while reclaim catches up can cost more than
> >   allowing reclaim parallelism to scale. It can also block higher-priority
> >   tasks in direct reclaim while they perform reclaim work on behalf of
> >   lower-priority memory-hogging tasks.
> >
> > - Concern: More kswapd threads may hide deeper reclaim issues.
> >   Response: This series is additive to ongoing reclaim improvements. In
> >   our testing, multi-threaded kswapd was able to improve performance on
> >   top of what multi-gen LRU already provides.
> >
> > - Concern: Existing knobs (such as swappiness and watermarks) should be
> >   preferred.
> >   Response: In our testing, those knobs alone did not reliably hit
> >   performance targets and could increase CPU cost for the same workload
> >   objective.
> >
> > - Concern: Need evidence from real workloads.
> >   Response: This cover letter includes Cassandra results showing higher
> >   throughput and lower response latency.
> >
> > - Concern: More reclaim threads may increase pressure on well-behaved
> >   tasks.
> >   Response: Adaptive wakeup addresses this by choosing thread count from
> >   node load.
> >
> > - Concern: Additional configuration can increase operational complexity.
> >   Response: The user-facing interface is intentionally minimal:
> >   max_kswapds_per_node.
> >
> > - Concern: Lock contention may serialize workers.
> >   Response: The Cassandra runs below still show net gains, indicating
> >   contention did not erase the benefit for this workload. The wakeup path
> >   now uses wake_up_nr() against the existing kswapd_wait queue, avoiding
> >   pgdat->kswapd_lock (a sleeping mutex) entirely on the allocation hot
> >   path.
> >
> > Real-world workload results (Cassandra):
> >
> > Tests were performed on 7.0.0-rc1.
> >
> > - max_kswapds_per_node=1
> >   - throughput sample: 171146
> >   - reference latency value: 6.375
> >   - op rates: 43256, 42731, 42341, 42818 ops/s
> >   - p99 latency: 6.3, 6.4, 6.4, 6.4 ms
> >
> > - max_kswapds_per_node=8
> >   - throughput sample: 183639
> >   - reference latency value: 6.0
> >   - op rates: 45791, 45253, 46534, 46061 ops/s
> >   - p99 latency: 6.0, 6.1, 5.9, 6.0 ms
> >
> > Observed improvement in these runs was about +7.3% throughput and
> > about -5.9% response latency, which shows practical benefit for
> > production-style database workloads.
> 
> But the result suggested we should not have multi-threaded kswapd, because
> 7 extra threads give only 7.3% throughput. That is not a good use of CPU
> cores. I suspect these threads are competing for LRU locks.
> We might want to partition each LRU list into N first, before N kswapds can
> work truely in parallel.
> 

Thanks for the feedback. I agree that scaling beyond a small number of
workers needs more investigation.

The purpose of this work is to make the maximum number of kswapd threads
configurable rather than hard-coding one thread per node. The useful
number is workload- and hardware-dependent, but the hard-coded value of
one is the least favorable value in all of our tests. The adaptive wakeup
policy only wakes workers when the node has idle CPU capacity. Therefore,
it does not force additional reclaim work onto already busy CPUs.

Improving P99 latency under memory pressure is difficult, so even a small
improvement can be valuable. These results also show that a value greater
than one can be beneficial for a given workload. In the latest test,
increasing from one to four threads improved throughput from 195,169 to
229,976 op/s (+17.8%) and reduced mean P99 latency from 6.300 ms to
5.975 ms (-5.2%).

Going from four to eight threads added only 0.8% throughput and regressed
P99 latency. This suggests that four threads are beneficial for this
workload, while higher counts need further investigation.

> >
> > In our runs, performance numbers were essentially unchanged with and
> > without the adaptive multi-threaded kswapd wakeup policy. In both
> > cases, they outperformed the single-kswapd-thread baseline. This
> > indicates the adaptive method preserved the multi-threaded performance
> improvement.
> >
> > Addendum: alternative approaches evaluated
> >
> > - PSI per NUMA node.
> >   I prototyped PSI-based node pressure ranges to drive wakeup count.
> >   This became cumbersome because robust PSI-to-thread mappings were
> not
> >   straightforward across workload types.
> >
> > - CPU mask snapshot policy.
> >   I also tested a simple CPU mask snapshot approach.
> >   While functional, it reflects a moment-in-time view and does not capture
> >   pressure trends over a broader sampling window.
> >
> > Buddy Lumpkin (1):
> >   vmscan: Support multiple kswapd threads per node
> >
> > Ehab Ababneh (2):
> >   mm/vmscan: handle racing max_seq advancement
> >   mm/vmscan: make kswapd wakeups NUMA load-aware
> >
> >  include/linux/mmzone.h        |   5 +-
> >  include/trace/events/vmscan.h |  28 +++
> >  mm/compaction.c               |   8 +-
> >  mm/internal.h                 |   3 +
> >  mm/page_alloc.c               |  26 +++
> >  mm/vmscan.c                   | 419
> +++++++++++++++++++++++++++++++++++++++---
> >  6 files changed, 465 insertions(+), 24 deletions(-)
> 
> 
> 
> 
> --
> Best Regards,
> Yan, Zi