[PATCH v2 0/5] mm: fix allocator races with memory hotplug

[Gregory Price (Meta)]

Five fixes for allocator paths that can act on state another CPU is
updating: zonelists rebuilt by memory hotplug, and the pagesets of a
zone that offlining empties.

I've packaged these together because they're all related to hotplug
in some way, but they may end up backporting differently.

Patch 1: the zonelist_update_seq retry is compiled out without
CONFIG_MEMORY_HOTREMOVE, but memory hot-add rewrites zonelists in place
too. A __GFP_THISNODE allocation that sees the list mid-rewrite fails
even though the node has free memory.

Patch 2: onlining a zone shifts zonelist slots under an in-flight walk,
so the allocator can use a zone its filters rejected: a GFP_KERNEL page
from ZONE_MOVABLE, which later makes offlining that block fail, or a
page from outside the nodemask. Without CONFIG_MEMORY_HOTREMOVE, it
relies on patch 1 to retry a walk that skips a zone and runs off the end
of the list.

Patch 3: offlining a zone's last block can move the zonelist's NULL
terminator onto an allocation's preferred slot, which the allocator then
dereferences in zone_statistics().

Patch 4: zone_pcp_reset() frees an emptied zone's pagesets while the
vmstat worker can still use them. The pagesets are now kept for the life
of the zone, which costs 320 bytes per possible CPU for each zone that
was onlined and then emptied.

Patch 5: __next_zones_zonelist() tests a slot's zone for NULL, then
reads the slot again to look up its node. If offlining turns the slot
into the terminator in between, zone_to_nid() dereferences NULL.
Current GCC and clang merge the two reads, but nothing requires it.

Reproducers
===========

All five races were reproduced on unmodified source in a QEMU guest
with three NUMA nodes:
   node 0: 3G, CPUs 0-1
   node 1: 1G ZONE_NORMAL, CPU 2
   node 2: 1G ZONE_NORMAL, CPU 3

And a 512M pc-dimm cold-plugged on node 1, left offline at boot
(CONFIG_ACPI_HOTPLUG_MEMORY=y).

Patch 1 needs CONFIG_MEMORY_HOTREMOVE=n to show the failure
Patch 5 needs mm/mmzone.o built with -fno-inline -fno-ipa-sra,
        without which the two reads are merged.

The windows are a few instructions wide, so I used a test module to
widen them with kprobes that stall one CPU inside the window. This
emulates an interrupt or vCPU preemption.  The vCPU preemption is the
more likely event to cause this at large scale with memory hotplug.

Patch 1 (zonelist):

  Online node 1's first memory block as ZONE_MOVABLE, which rebuilds
  node 1's zonelists.

  Stall the rebuild after it writes ZONE_MOVABLE into the first entry
  but before it writes ZONE_NORMAL into the second.

  On another CPU, allocate GFP_KERNEL | __GFP_THISNODE from node 1.

  Before: the allocation fails, 3/3 runs.
  After:  the allocation waits for the rebuild and succeeds, 3/3 runs.

Patch 2 (zonelist slots):

  On one CPU, allocate GFP_KERNEL with a nodemask, and stall it after
  the iterator approves its first zonelist slot but before the
  allocator loads the zone from that slot.

  Online node 1's first memory block as ZONE_MOVABLE, which inserts
  ZONE_MOVABLE at the front of node 1's zonelists.

  Before: nodemask {1} gets a node 1 ZONE_MOVABLE page and nodemask
          {2} a node 1 ZONE_NORMAL page, for single-page and bulk
          allocations, 3/3 runs each.
  After:  nodemask {1} gets node 1 ZONE_NORMAL and nodemask {2} gets
          node 2 ZONE_NORMAL, 3/3 runs each.

Patch 3 (preferred zone):

  Online node 1's block as ZONE_MOVABLE, so node 1's local zonelist is
  [MOVABLE, NORMAL].

  On another CPU, allocate GFP_KERNEL | __GFP_THISNODE from node 1,
  whose preferred slot is NORMAL, and stall it after it loads the zone.

  Offline the block, which rebuilds the list as [NORMAL] and moves the
  terminator into the preferred slot.

  Before: NULL dereference in get_page_from_freelist(), 3/3 runs.
  After:  the allocation succeeds from ZONE_NORMAL, 3/3 runs.

Patch 4 (pagesets):

  Online node 1's block as ZONE_MOVABLE, then offline it.

  Stall the offline before the zone empties, and stall the vmstat
  worker on another CPU (kicked through /proc/sys/vm/stat_refresh) in
  decay_pcp_high() for that zone. Let the offline free the pagesets,
  reallocate the freed percpu area with a known value, then release
  the worker.

  Before: decay_pcp_high() runs on the freed pageset, 3/3 runs. In
          1/3 the area was reallocated before the worker resumed, and
          decay_pcp_high() rewrote the new object.
  After:  the pageset is not freed, 3/3 runs.

  Repeated offline/online of the block, and ejecting and re-adding the
  only DIMM of a CPU-less node over QMP, reuse the same pagesets.

Patch 5 (slot reread):

  Online node 1's block as ZONE_MOVABLE, so node 1's local zonelist is
  [MOVABLE, NORMAL].

  On another CPU, allocate GFP_KERNEL | __GFP_THISNODE from node 1
  with nodemask {1}, and stall it after __next_zones_zonelist() tests
  the NORMAL slot for NULL but before it reads the slot again.

  Offline the block, which rebuilds the list as [NORMAL] and moves the
  terminator into that slot.

  Before: NULL dereference in zone_to_nid(), 4/4 runs.
  After:  no oops, 3/3 runs.

All five reproducers also pass on the series tip, 3/3 runs each.

Changes in v2
=============
- Added patches 2-5.
- Pulled cpuset patch out by itself

Link: https://lore.kernel.org/linux-mm/20261014222456.2318284-1-gourry@xxxxxxxxxx/
Link: https://lore.kernel.org/linux-mm/20261015210539.3243563-1-gourry@xxxxxxxxxx/

Gregory Price (Meta) (5):
  mm/page_alloc: detect zonelist rebuilds from memory hot-add
  mm/page_alloc: validate zonelist slots against concurrent rebuilds
  mm/page_alloc: snapshot the preferred zone against shrink rebuilds
  mm/page_alloc: keep a zone's pagesets when it goes empty
  mm/mmzone: read each zoneref slot once when filtering by nodemask

 mm/mmzone.c     | 11 +++----
 mm/page_alloc.c | 77 +++++++++++++++++++++++++++++++++++++------------
 2 files changed, 65 insertions(+), 23 deletions(-)

-- 
2.53.0-Meta