[RFC PATCH 0/1] mm/swap: keep reclaim batches contiguous on swap

[Alexandre Ghiti]

7.3 regressed swap I/O: for the same swap volume, a workload that
rewrites swapped out memory sends many more requests to the device.
mysql/sysbench in a 256M memcg swapping to an NVMe partition, 7.3
compared to the previous version:

  device read requests per transaction        +59%
  device write requests per transaction       +27%
  throughput, iocost off                      no change
  throughput, iocost on                       -19%

Two commits interact:

- Commit 9a42aed48421 ("mm, swap: prefer nonfull over free clusters"),
  in 6.18, makes order-0 allocations fill nonfull clusters before
  taking a free one.

- Commit 3cedb745d354 ("mm: clarify the folio_free_swap() for
  do_swap_page()"), in 7.3, frees the swap slot on an exclusive write
  fault. 7.0 to 7.2 kept the slot, but only as a side effect of commit
  4b34f1d82c65 ("mm, swap: free the swap cache after folio is
  mapped").

The freed slots leave holes in full clusters, which then go back on
the nonfull list. Reclaim allocates slots one folio at a time and
merges the writes afterwards, so a batch of LRU-adjacent folios gets
scattered over those holes and is written with many small requests.
Readahead later finds those pages scattered as well, and needs more
read requests to bring them back.

I tried three fixes, which perform similarly:

A. Revert 3cedb745d354. This keeps the slot on write fault, but undoes
   the intent of that commit and keeps swap space pinned under dirty
   pages.

B. Below 50% swap usage, make order-0 allocations take a free cluster
   before a nonfull one. This is what the allocator already does, at
   any usage, on swap devices with discard enabled (swapon
   --discard=pages). It is about five lines, but the threshold is
   arbitrary, it stops working as swap fills up, and it applies to
   every order-0 allocation, using up the free clusters that
   nonfull-first keeps for mTHP.

C. This patch. Reclaim tells the allocator how many pages of each
   small order the batch is about to swap out, and those allocations
   then follow contiguous free slots, in a nonfull cluster if one has
   them and in a free cluster otherwise. Nothing is reserved, there
   is no threshold, and nonfull-first is kept.

Same workload as above, 7.3 with C relative to 7.3:

  device read requests per transaction        -12%
  device write requests per transaction       -36%
  throughput, iocost off                      no change
  throughput, iocost on                       +41%

I think C is the way to go. I just add the patch to this regression
notice so that we have an idea of what it takes to implement, the
implementation may not be perfect.

I would welcome other ideas.

Alexandre Ghiti (1):
  mm/swap: allocate contiguous swap slots for a reclaim batch

 mm/shmem.c    |   2 +
 mm/swap.h     |  10 +++
 mm/swapfile.c | 211 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 mm/vmscan.c   |  11 ++-
 4 files changed, 229 insertions(+), 5 deletions(-)

base-commit: fcec4f4bcc1101dbd7e3aff267856b3bbfe128c6