[PATCH v6 0/7] mm/memory-failure: keep hardware-poisoned pages out of the next kexec

[Breno Leitao]

Problem:
========

When a page is hard-offlined due to an uncorrectable memory error (multi
bit ECC), memory_failure() sets PG_hwpoison, unmaps it, removes it from
the buddy allocator. This information is not carried to the next kernel
that is kexec'd. The kexec'd kernel then trips over that bad memory
bank _again_.

Why now:
========

Several industry trends make this increasingly important:

    1) DRAM is getting more expensive
    2) soldered / on-package memory (LPDDR, HBM) is becoming more common, so a
       failing part can no longer simply be swapped;
    3) memory is kept in service far longer (at Meta, DRAM lifetime is being
       drastically extended).
    4) It is more and more common to kexec instead of full reboot
    5) Increase of memory per system with CXL

What was done already:
======================

To deal with the problem above, I have already fixed a few things in
this area:

1) Panic on unrecoverable errors, instead of "printk and carry over":
https://lore.kernel.org/all/20260630-ecc_panic-v10-0-c6ed5b62eea2@xxxxxxxxxx/

2) Respect poisoned memory at kexec time
https://lore.kernel.org/all/20260812-kexec_posioned-v6-0-e477887086f0@xxxxxxxxxx/

Now, the natural follow up is to carry the poisoned memory information
to the next kexec kernel, avoiding tripping over a known "bad page".

Proposed Solution:
==================

Carry the poisoned frames to the next kernel in a new EFI configuration
table, LINUX_EFI_POISONED_MEMORY.

The table is a bitmap with one bit per 2MB of physical memory, modeled
on LINUX_EFI_UNACCEPTED_MEMORY. The stub sizes it from the EFI memory
map and installs it empty while boot services are up -- a running kernel
cannot install a configuration table, it can only flip bits -- and a
table that an earlier stub run in the same firmware session installed
is reused as-is. That is one fixed-size allocation, 64KB per TiB of the
span the memory map describes, with no list to grow at runtime and no
chain to trust at parse time.

The mechanism is architecture independent, so x86 and arm64 use the same
code.

Each hard offline sets the bit for its unit. Soft-offlined pages are not
recorded: they are still functional, and were offlined predictively.
Errors injected by software (MADV_HWPOISON, hwpoison-inject) are not
recorded either: they are not hardware faults.

The next kernel takes the table into use from efi_config_parse_tables().
It vets the inherited header and reserves the table's own pages in
memblock.

Then, once the architecture has reserved the kernel image and the
initrd, it reads the bitmap once and reserves every recorded unit in
memblock with a new MEMBLOCK_RSRV_HWPOISON flag. On x86 and arm64 this
runs before the page tables and the memmap are allocated, so memblock
cannot hand a recorded unit to an early allocation, and it never frees
one to the buddy allocator.

memmap_init_reserved_pages() then flags the frames of those units
PG_hwpoison, initializing their struct pages through init_deferred_page()
where that initialization is deferred. They end up in the state a frame
poisoned by this kernel would be in, so everything that already
understands PG_hwpoison covers them -- including the kexec segment
placement check from the series linked above, which keeps the segments a
kexec places off these frames. From there on the memmap is the only
source of poisoned frames: nothing reads the bitmap again.

The memory block poison counter is seeded from the memmap when a block is
created at boot, behind num_poisoned_pages.

Granularity is the trade-off: one bad 4KB frame costs a whole 2MB unit in
every later kernel of the chain. In exchange, a row or column fault --
roughly a quarter of the DRAM faults reported in [1], and potentially
thousands of 4KB pages scattered over gigabytes -- collapses into a bit
or two.

A bit is never cleared, which is a known limitation: it stands for a
whole unit, so an unpoison of one frame cannot tell whether the unit as a
whole is good again.

Known limitations:
==================

In order to keep this patchset digestible, I am making some trade-offs,
thus, this feature has the following limitations:

  - Memory hot-added after boot is not covered: the bitmap spans the RAM
    the EFI memory map describes, and a frame above it is not recorded.
    Same gap the unaccepted-memory table has.

  - The bitmap spans the descriptors that are write-back cacheable or
    unaccepted memory. On x86 the stub also reports loader, boot services
    and conventional memory without EFI_MEMORY_WB as RAM, so such a
    descriptor at either end of RAM falls outside the bitmap and its
    frames are not recorded.

  - A recorded unit that overlaps a reservation made before the units are
    reserved, the kernel image or a kexec segment for one, is flagged only
    where that reservation leaves it free, and the overlap is logged. A
    kexec segment was placed around the bad frames, so the frames it
    leaves free include them. A unit that is covered entirely is skipped
    with a warning.

  - The reserved array cannot be resized when the table is parsed, so
    memblock_reserve_hwpoison() keeps 32 slots free for the reservations
    that follow and skips the units that do not fit, with a warning.
    Adjacent units share a slot.

  - Memory preserved across a KHO handover is not covered.
    kho_preserved_memory_reserve() reserves it after the units are
    reserved and marks it MEMBLOCK_RSRV_NOINIT, so
    memmap_init_reserved_pages() skips it and a recorded frame there is
    not flagged. The frame goes back to the allocator once the owner
    releases it.

  - On x86 the next kernel picks its own physical address again.
    choose_random_location() draws from the memory map and does not know
    about the table, so KASLR can land the kernel image on a recorded
    frame. arm64 does not re-pick; the outgoing kernel fixes the address.

  - An inherited frame keeps its direct map entry. On x86 the MCE handler
    calls set_mce_nospec() next to memory_failure(), and nothing replays
    that half here, so the frame is out of the allocator but still mapped
    writeback. set_memory_*() needs the allocator to split a large page,
    so this wants a later pass.

  - On a host with SME active the frames are not recorded: the stub writes
    the table in the clear, so reading it back there needs a decrypted
    mapping. Parsing an inherited table is not affected. Mapping the table
    decrypted is follow up work.

  - The CMA patchset is being worked by Shaikh Kamaluddin
    https://lore.kernel.org/all/20261010162017.62506-1-shaikhkamal2012@xxxxxxxxx/#t

All of the limitations above can be fixed in follow up work. I am trying
to keep this patchset the foundation, with that work done on top.

The series is seven patches:

  1. Add the table: Defines the EFI table: GUID, Kconfig symbol, and
     a bitmap with one bit per 2MiB.
  2. Install it from the stub: The EFI stub allocates the table at boot,
     sized from the memory map, installed empty.
  3. Flag an inherited frame: Adds a helper flagging an inherited frame
     PG_hwpoison, counting it in the global counter and the managed pages,
     and disabling unpoison.
  4. Memblock flag: Adds a memblock flag keeping poisoned ranges from
     allocators and buddy, and marking frames PG_hwpoison.
  5. Memory block counter: Seeds each memory block's poison counter from
     the memmap at boot, only when poison exists.
  6. Reserve the inherited units: Reads the inherited table once at boot
     and reserves every recorded 2MiB unit in memblock.
  7. Record poisoned frames: Records each hard-offlined frame's unit in
     the table, so the next kexec'd kernel reserves it.

This was initially discussed at
https://lore.kernel.org/all/ajut_LDQGYCShApx@xxxxxxxxx/

A special thanks to Kiryl and David, for feedbacks and suggestions.

[1] https://arxiv.org/abs/2408.15302

To: Ard Biesheuvel <ardb@xxxxxxxxxx>
To: Ilias Apalodimas <ilias.apalodimas@xxxxxxxxxx>
To: Miaohe Lin <linmiaohe@xxxxxxxxxx>
To: Naoya Horiguchi <nao.horiguchi@xxxxxxxxx>
To: Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx>
Cc: linux-efi@xxxxxxxxxxxxxxx
Cc: linux-kernel@xxxxxxxxxxxxxxx
Cc: linux-mm@xxxxxxxxx
Cc: rmikey@xxxxxxxx
To: kas@xxxxxxxxxx
Cc: riel@xxxxxxxxxxx
To: kexec@xxxxxxxxxxxxxxxxxxx

Signed-off-by: Breno Leitao <leitao@xxxxxxxxxx>
---
Changes in v6:
- Rework the consumer along the lines David Hildenbrand described: read the
  bitmap once, at parse time, and reserve each recorded unit in memblock
  with a new MEMBLOCK_RSRV_HWPOISON flag. memmap_init_reserved_pages()
  flags the frames PG_hwpoison, so the memmap is the only source of
  poisoned frames from then on. memblock never allocates from a unit or
  frees it to buddy, which also keeps early allocations off it (David)
  * This was discussed in the MM alignement meeting/thread as well.
- Drop the __free_pages_core() hook, the accept-and-free refactor under it
  and the range query. Nothing queries the bitmap after parse time (David)
- Seed the memory block poison counter from the memmap, behind
  num_poisoned_pages, instead of walking the bitmap (David)
- Record a hard error on a frame that was soft-offlined earlier: the hook
  moves ahead of the MF_MSG_ALREADY_POISONED statistics guard (Sashiko)
- Do not count inherited frames in the per-node memory failure statistics:
  this kernel handled no failure
- Reject a table whose units would run past the address space
- Do not record on a host with SME active: the table is written in the
  clear and needs a decrypted mapping
- Do not record errors injected by software (MF_SW_SIMULATED): they are
  not hardware faults, and a recorded bit is never cleared
- Map the table with memremap() for recording instead of using the direct
  map, which mem= and a crash kernel's usable range trim on arm64
- Reserve the table's pages without adding them to memblock.memory: they
  are no longer read through the direct map, and with mem= the extra
  region made a CONFIG_DEBUG_VM kernel warn
- Treat an inherited frame as a hardware failure, so unpoison_memory()
  cannot free it, and count it as managed so that memory offlining adds up
- Link to v5: https://patch.msgid.link/20260915-hwpoison-kho-v5-0-3bc7a57bd503@xxxxxxxxxx

Changes in v5:
- Free the part of the block that is not recorded instead of dropping the
  block whole: the leftover frames were neither buddy nor poisoned nor
  offline, so they leaked and left the block impossible to offline
  (Sashiko)
- Count inherited frames into the memory block when it is created.
  memblk_nr_poison_inc() cannot run that early, and unpoisoning a frame
  whose block counter was never incremented wrapped it to ULONG_MAX,
  which then refuses memory_block_online() for good (Sashiko)
- Keep the caller's end in range_contains_poisoned_memory() when the
  start is clamped up to phys_base, rather than shifting the whole window
  past what was asked about (Sashiko)
- Reject an inherited table whose footprint wraps the address space,
  which fed a negative range to memblock_add() (Sashiko)
- Say in patch 2 why the native bitops are safe: EFI_STUB && 64BIT is
  little-endian everywhere (Sashiko)
- Split the accept-and-free tail of __free_pages_core() into its own
  preparation patch at the head of the series, so the replay patch can
  reuse it without carrying a rework of that function
- Split the replay itself into the page flag helper, the memory block
  counter and the __free_pages_core() hook, one per subsystem it touches
- Split adopting an inherited table from recording into it: the parse-time
  plumbing and the memory_failure() hook are unrelated pieces of work
- Put the memory block counter ahead of the __free_pages_core() hook, so
  no revision in the middle of the series flags a frame the counter does
  not know about
- Size the bitmap from the descriptors that become RAM rather than from
  every descriptor in the map, which v4 argued for. Both architectures
  gate on EFI_MEMORY_WB, so spanning that plus EFI_UNACCEPTED_MEMORY
  stays wider than either accepts on its own while leaving out the MMIO
  apertures. On a 5G guest the table went from 65536 to 320 bytes
- Link to v4: https://patch.msgid.link/20260909-hwpoison-kho-v4-0-359313564495@xxxxxxxxxx

Changes in v4:
- Poison the inherited frames from __free_pages_core(), as each block
  reaches the buddy allocator, instead of walking the bitmap once from
  mm_core_init(): with CONFIG_DEFERRED_STRUCT_PAGE_INIT most struct pages
  are not initialised there and the poison was silently lost (Kiryl)
- Hand the table's pages to memblock at parse time, the way
  8dbe33956d96 does for the unaccepted memory table; it is ACPI reclaim
  memory and touching it through the direct map faulted
- Add phys_base to the table, so a machine whose RAM starts high does not
  pay for the hole below it (Kiryl)
- Size the bitmap from every descriptor in the memory map rather than a
  descriptor-type list, which was x86-only reasoning (Kiryl)
- Drop the max_pfn clamp on the inherited bitmap size (Kiryl)
- Drop the stale memblock.h include (Kiryl)
- Link to v3: https://patch.msgid.link/20260826-hwpoison-kho-v3-0-6f79c4b605bc@xxxxxxxxxx

Changes in v3:
- Poison the inherited frames from mm_core_init() instead of reserving
  them in memblock, so everything that keys off PG_hwpoison sees them,
  the kexec segment placement check included (Kiryl)
- Reserve nothing in memblock: the page flag is enough, and reserving
  per unit that early runs into the fixed region array (Kiryl)
- Drop the 1MB cap on the table and the unit coarsening that went with
  it (Kiryl)
- Size the table from the memory types arm64 turns into RAM as well, not
  just the set setup_e820() maps to E820_TYPE_RAM (Kiryl)
- Make CONFIG_EFI_POISONED_MEMORY unprompted, so it is on wherever its
  dependencies allow and nobody has to decide (Kiryl, Pratyush)
- Fold the top-of-RAM helper into its only caller and make it static,
  rather than adding a generic libstub API
- Fold the table build and its installation into one patch
- Spell out the known limitations in this cover letter
- Link to v2: https://patch.msgid.link/20260821-hwpoison-kho-v2-0-5743791e48e6@xxxxxxxxxx

Changes in v2:
- Replace the growable linked list of 4KB entries with a fixed-size
  bitmap, one bit per 2MB, modeled on the unaccepted-memory table (Kiryl)
- Record hard offlines only, by hooking action_result() instead of
  num_poisoned_pages_inc(), which also fires for soft offline (Kiryl)
- Allocate the table as EFI_ACPI_RECLAIM_MEMORY, so it is not System RAM
  in the next kernel, and reuse an inherited table instead of installing
  a second one
- Validate the geometry of an inherited table before using it
- Cap the table at 1MB, coarsening the unit instead of growing it
- Restrict to 64-bit, as the unaccepted-memory table effectively is
- Never clear a bit: an unpoison no longer un-records the unit
- Split the table definition and the stub installer into separate patches
- Link to v1: https://patch.msgid.link/20260717-hwpoison-kho-v1-0-9c5eda551998@xxxxxxxxxx

To: Ard Biesheuvel <ardb@xxxxxxxxxx>
To: Ilias Apalodimas <ilias.apalodimas@xxxxxxxxxx>
To: Miaohe Lin <linmiaohe@xxxxxxxxxx>
To: Naoya Horiguchi <nao.horiguchi@xxxxxxxxx>
To: Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx>
To: David Hildenbrand <david@xxxxxxxxxx>
To: Lorenzo Stoakes <ljs@xxxxxxxxxx>
To: "Liam R. Howlett" <liam@xxxxxxxxxxxxx>
To: Vlastimil Babka <vbabka@xxxxxxxxxx>
To: Mike Rapoport <rppt@xxxxxxxxxx>
To: Suren Baghdasaryan <surenb@xxxxxxxxxx>
To: Michal Hocko <mhocko@xxxxxxxx>
To: Thomas Gleixner <tglx@xxxxxxxxxx>
To: Ingo Molnar <mingo@xxxxxxxxxx>
To: Borislav Petkov <bp@xxxxxxxxx>
To: Dave Hansen <dave.hansen@xxxxxxxxxxxxxxx>
To: x86@xxxxxxxxxx
To: "H. Peter Anvin" <hpa@xxxxxxxxx>
To: Brendan Jackman <brendan.jackman@xxxxxxxxx>
To: Johannes Weiner <hannes@xxxxxxxxxxx>
To: Zi Yan <ziy@xxxxxxxxxx>
To: Oscar Salvador <osalvador@xxxxxxx>
To: Greg Kroah-Hartman <gregkh@xxxxxxxxxxxxxxxxxxx>
To: "Rafael J. Wysocki" <rafael@xxxxxxxxxx>
To: Danilo Krummrich <dakr@xxxxxxxxxx>
Cc: linux-efi@xxxxxxxxxxxxxxx
Cc: linux-kernel@xxxxxxxxxxxxxxx
Cc: linux-mm@xxxxxxxxx
Cc: harry@xxxxxxxxxx
Cc: linux-cxl@xxxxxxxxxxxxxxx
Cc: driver-core@xxxxxxxxxxxxxxx
To: shakeel.butt@xxxxxxxxx

---
Breno Leitao (7):
      mm/memory-failure: efi: add the LINUX_EFI_POISONED_MEMORY configuration table
      mm/memory-failure: libstub: install the poisoned-memory EFI table
      mm/memory-failure: add hwpoison_boot_page() to flag an inherited frame
      mm/memblock: add a reserved-region flag for poisoned frames
      drivers/base/memory: count the poison a block already holds when creating it
      mm/memory-failure: efi: reserve the inherited poisoned-memory units
      mm/memory-failure: efi: record hardware-poisoned frames into the poisoned-memory table

 arch/arm64/mm/init.c                           |   3 +
 arch/x86/kernel/setup.c                        |   3 +
 arch/x86/platform/efi/efi.c                    |   3 +
 drivers/base/memory.c                          |  30 +++
 drivers/firmware/efi/Kconfig                   |   8 +
 drivers/firmware/efi/Makefile                  |   1 +
 drivers/firmware/efi/efi.c                     |   8 +
 drivers/firmware/efi/libstub/efi-stub-helper.c | 103 +++++++++++
 drivers/firmware/efi/libstub/efi-stub.c        |   1 +
 drivers/firmware/efi/libstub/efistub.h         |   6 +
 drivers/firmware/efi/libstub/x86-stub.c        |   2 +
 drivers/firmware/efi/poison.c                  | 243 +++++++++++++++++++++++++
 include/linux/efi.h                            |  24 +++
 include/linux/memblock.h                       |  10 +
 mm/internal.h                                  |   5 +
 mm/memblock.c                                  |  80 +++++++-
 mm/memory-failure.c                            |  61 ++++---
 17 files changed, 567 insertions(+), 24 deletions(-)
---
base-commit: a9d7ced84989ec05be09b4b8428759ef60450a0f
change-id: 20260622-hwpoison-kho-fc9db2ada8ba

Best regards,
--  
Breno Leitao <leitao@xxxxxxxxxx>