Unknown NMI after resume from hibernation: restore kernel leaves the NMI watchdog counter running on CPU 0

[Sergei]

Hi,

I'm hitting the problem Tony reported in July 2024 [1] on an AMD laptop,
and as far as I can tell it's still there in current mainline, so here's
another data point.

After almost every resume from hibernation I get exactly one

  Uhhuh. NMI received for unknown reason 2c on CPU 0.
  Dazed and confused, but trying to continue

somewhere between 1 and ~130 seconds after "hibernation exit". It shows
up after 25 of the last 28 hibernation cycles, always on CPU 0, and never
after suspend-to-RAM.

Machine: Huawei NBLL-WXX9, AMD Ryzen 5 4500U (Renoir, family 0x17),
BIOS 1.07. Seen on 7.2.3 and reproduced on 7.3-rc1 (both Manjaro builds;
the only taint is U, from ryzenadj writing to PCI config space). TLP is
installed and sets kernel.nmi_watchdog=0 at runtime (that's its default);
the kernel command line doesn't touch the watchdog, so it is on at boot.

It's the same sequence as in [1]: the restore kernel boots with the NMI
watchdog enabled, programs the perf counter on every CPU, takes the APs
down and jumps into the image. The image kernel has the watchdog off, so
nothing owns the counter that is still running on CPU 0. When it
overflows, perf_event_nmi_handler() finds no active events and the NMI
ends up as unknown. After the wrap the counter just keeps counting from
zero, which is why there is only one NMI per resume.

Reading the PMU MSRs right after a resume on 7.3-rc1 shows it directly:

  CPU0:    PERF_CTL0 = 0x530076  PERF_CTR0 = 0xfffe53fa3a48  (EN, INT)
  CPU1..5: PERF_CTL0 = 0x130076  PERF_CTR0 = 0xfffa...-0xfffe...  (INT only)

Event 0x76 with the counters sitting at about -(watchdog_thresh * cpu_khz)
(23.7e9 here) is the watchdog's setup. CPU0 had about 7.2e9 cycles left,
and the unknown NMI came 69 seconds after "hibernation exit".

Booting with nmi_watchdog=0, so the restore kernel doesn't arm it either,
makes it go away: after a hibernation cycle every PERF_CTLn is 0 on all
CPUs and no NMI shows up.

I don't see anything in the resume path that stops the hardlockup
detector before swsusp_arch_resume() (resume_target_kernel() only touches
the softlockup watchdog), and the two patches from [1] never got a reply.
Laptops running TLP probably all see this, since turning the NMI watchdog
off is TLP's default.

Is the direction Pavel suggested back then (suspend/resume hooks in the
watchdog code) still the preferred one, or should the restore kernel just
stop the hardlockup detector before jumping into the image? I'm happy to
test patches.

[1] https://lkml.iu.edu/hypermail/linux/kernel/2407.3/01199.html

Thanks,
Sergei Gorbunov