Re: hunting memory corruption bug in 6.18.x
From: Borislav Petkov
Date: Wed Oct 14 2026 - 19:55:44 EST
On Mon, Oct 12, 2026 at 01:43:30PM +0200, Nikola Ciprich wrote:
> Some more news.
>
> In the meantime, we’ve had the same crash on another of our own
> cluster nodes. This time it was booted with tlbi=ipi, so that
> didn’t help either, exactly as Borislav said.
>
> (Just a side note: it seems quite hard to check which TLB
> invalidation mechanism is in use. There is no message in dmesg,
> and invlpgb is deliberately not displayed in the /proc/cpuinfo flags,
> so apart from measuring with perf, I simply have to trust that
> the kernel parameter does what it should for my kernel release.)
Yeah, we probably should do something there. Maybe show it in /proc/cpuinfo as
"tlbi" when enabled.
diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h
index f62944700975..44ecc82f2ca0 100644
--- a/arch/x86/include/asm/cpufeatures.h
+++ b/arch/x86/include/asm/cpufeatures.h
@@ -342,7 +342,7 @@
#define X86_FEATURE_CLZERO (13*32+ 0) /* "clzero" CLZERO instruction */
#define X86_FEATURE_IRPERF (13*32+ 1) /* "irperf" Instructions Retired Count */
#define X86_FEATURE_XSAVEERPTR (13*32+ 2) /* "xsaveerptr" Always save/restore FP error pointers */
-#define X86_FEATURE_INVLPGB (13*32+ 3) /* INVLPGB and TLBSYNC instructions supported */
+#define X86_FEATURE_INVLPGB (13*32+ 3) /* "tlbi" INVLPGB and TLBSYNC instructions supported */
#define X86_FEATURE_RDPRU (13*32+ 4) /* "rdpru" Read processor register at user level */
#define X86_FEATURE_WBNOINVD (13*32+ 9) /* "wbnoinvd" WBNOINVD instruction */
#define X86_FEATURE_AMD_IBPB (13*32+12) /* Indirect Branch Prediction Barrier */
> So if I understand it correctly, we can now completely rule out
> an AMD firmware bug, right?
>
> Since I didn’t have anything else to try, and we started having
> crashes on our own cluster nodes, where I can be a bit bolder,
> I built 7.3-rc6, deployed it on two problematic AMD nodes, and
> started a migration torture loop.
>
> Over the weekend, there were ~70 migrations of ~160 VMs between
> those two nodes without a single issue.
>
> Today I upgraded both boxes to rc7 and started the migrations again.
>
> I can’t be sure yet, but it really seems the problems are gone in 7.3.
>
> The question is: what now? (I still think bisecting would be unreliable.)
>
> For those interested, I’ve tidied up the shared files a little and
> added some more content to https://storage.linuxbox.cz/index.php/s/6rcp8oWCGzAqPEY:
So I took a look. The one you added the sources for is this: 20261008-lbxv1f
and below is what I could decipher it to.
Long story short, I *think* that dentry pointer becomes funky:
hlist_bl_for_each_entry_rcu(dentry, node, b, d_hash) {
unsigned seq;
/*
* The dentry sequence count protects us from concurrent
* renames, and thus protects parent and name fields.
*
* The caller must perform a seqcount check in order
* to do anything useful with the returned dentry.
*
* NOTE! We do a "raw" seqcount_begin here. That means that
* we don't wait for the sequence count to stabilize if it
* is in the middle of a sequence change. If we do the slow
* dentry compare, we will do seqretries until it is stable,
* and if we end up with a successful lookup, we actually
* want to exit RCU lookup anyway.
*
* Note that raw_seqcount_begin still *does* smp_rmb(), so
* we are still guaranteed NUL-termination of ->d_name.name.
*/
seq = raw_seqcount_begin(&dentry->d_seq);
^^^^^^^^^^^^^^
Now, the $10^6 question is, do you have the crash dump still and if so, can
you figure out what the real address of that dentry is?
The wrong one is: 0xffffffff0c930020 so I'm guessing that 0 after the ffff's
should actually be 81-something as that's kernel memory.
Hmmm?
Thx.
[11402.940943] BUG: unable to handle page fault for address: ffffffff0c93001c
[11402.942273] #PF: supervisor read access in kernel mode
[11402.943629] #PF: error_code(0x0000) - not-present page
[11402.945025] PGD 6d6f83a067 P4D 6d6f83b067 PUD 0
[11402.946469] Oops: Oops: 0000 [#1] SMP NOPTI
[11402.947950] CPU: 23 UID: 0 PID: 704950 Comm: servercare-moni Kdump: loaded Tainted: G E 6.18.55lb9.01 #1 PREEMPT(voluntary)
[11402.951254] Tainted: [E]=UNSIGNED_MODULE
[11402.952913] Hardware name: ASUSTeK COMPUTER INC. RS720A-E12-RS12/K14PP-D24 Series, BIOS 1201 08/25/2023
[11402.956569] RIP: 0010:__d_lookup_rcu+0x4d/0xe0
ffffffff8165ae30 <__d_lookup_rcu>
+0x4d = 0xffffffff8165ae7d
[11402.958452] Code: 48 8d 04 c2 f6 07 02 0f 85 a0 00 00 00 48 8b 10 48 89 d0 48 83 e0 fe 48 83 fa 01 77 0d e9 80 00 00 00 48 8b 00 48 85 c0 74 78 <44> 8b 58 fc 48 39 78 10 75 ee 48 83 78 08 00 74 e7 48 39 58 18 75
[11402.958452] Code: 48 8d 04 c2 f6 07 02 0f 85 a0 00 00 00 48 8b 10 48 89 d0 48 83 e0 fe 48 83 fa 01 77 0d e9 80 00 00 00 48 8b 00 48 85 c0 74 78 <44> 8b 58 fc 48 39 78 10 75 ee 48 83 78 08 00 74 e7 48 39 58 18 75
ffffffff8165ae6e: 77 0d ja ffffffff8165ae7d <__d_lookup_rcu+0x4d>
ffffffff8165ae70: e9 80 00 00 00 jmp ffffffff8165aef5 <__d_lookup_rcu+0xc5>
ffffffff8165ae75: 48 8b 00 mov (%rax),%rax
ffffffff8165ae78: 48 85 c0 test %rax,%rax
ffffffff8165ae7b: 74 78 je ffffffff8165aef5 <__d_lookup_rcu+0xc5>
ffffffff8165ae7d: 44 8b 58 fc mov -0x4(%rax),%r11d
<--- here
that looks like:
struct dentry *__d_lookup_rcu(const struct dentry *parent,
...
seq = raw_seqcount_begin(&dentry->d_seq);
...
#define raw_read_seqcount(s) \
({ \
unsigned __seq = seqprop_sequence(s); \
which is that unreadable gunk:
# ./include/linux/seqlock.h:227: SEQCOUNT_LOCKNAME(spinlock, spinlock_t, __SEQ_RT, spin)
#NO_APP
movl -4(%rax), %r11d # MEM[(const volatile unsigned int *)node_49 + -4B], _25
# ./include/linux/seqlock.h:227: SEQCOUNT_LOCKNAME(spinlock, spinlock_t, __SEQ_RT, spin)
i.e., that dentry->d_seq _Generic selector pointer thing:
seq = ({ ({ unsigned __seq =
_Generic(*(&dentry->d_seq), seqcount_t:
__seqprop_sequence, seqcount_raw_spinlock_t:
__seqprop_raw_spinlock_sequence, seqcount_spinlock_t:
__seqprop_spinlock_sequence, seqcount_rwlock_t:
__seqprop_rwlock_sequence, seqcount_mutex_t:
__seqprop_mutex_sequence)(&dentry->d_seq);
kcsan_atomic_next(1000); __seq; }) & ~1; });
ffffffff8165ae81: 48 39 78 10 cmp %rdi,0x10(%rax)
ffffffff8165ae85: 75 ee jne ffffffff8165ae75 <__d_lookup_rcu+0x45>
ffffffff8165ae87: 48 83 78 08 00 cmpq $0x0,0x8(%rax)
ffffffff8165ae8c: 74 e7 je ffffffff8165ae75 <__d_lookup_rcu+0x45>
ffffffff8165ae8e: 48 39 58 18 cmp %rbx,0x18(%rax)
ffffffff8165ae92: 75 e1 jne ffffffff8165ae75 <__d_lookup_rcu+0x45
All code
========
0: 48 8d 04 c2 lea (%rdx,%rax,8),%rax
4: f6 07 02 testb $0x2,(%rdi)
7: 0f 85 a0 00 00 00 jne 0xad
d: 48 8b 10 mov (%rax),%rdx
10: 48 89 d0 mov %rdx,%rax
13: 48 83 e0 fe and $0xfffffffffffffffe,%rax
17: 48 83 fa 01 cmp $0x1,%rdx
1b: 77 0d ja 0x2a
1d: e9 80 00 00 00 jmp 0xa2
22: 48 8b 00 mov (%rax),%rax
25: 48 85 c0 test %rax,%rax
28: 74 78 je 0xa2
2a:* 44 8b 58 fc mov -0x4(%rax),%r11d <-- trapping instruction
rax = ffffffff0c930020
2e: 48 39 78 10 cmp %rdi,0x10(%rax)
32: 75 ee jne 0x22
34: 48 83 78 08 00 cmpq $0x0,0x8(%rax)
39: 74 e7 je 0x22
3b: 48 39 58 18 cmp %rbx,0x18(%rax)
3f: 75 .byte 0x75
Code starting with the faulting instruction
===========================================
0: 44 8b 58 fc mov -0x4(%rax),%r11d
4: 48 39 78 10 cmp %rdi,0x10(%rax)
8: 75 ee jne 0xfffffffffffffff8
a: 48 83 78 08 00 cmpq $0x0,0x8(%rax)
f: 74 e7 je 0xfffffffffffffff8
11: 48 39 58 18 cmp %rbx,0x18(%rax)
15: 75 .byte 0x75
[11402.964498] RSP: 0018:ff3eefdafcacfc40 EFLAGS: 00010292
[11402.966642] RAX: ffffffff0c930020 RBX: 000000061356452c RCX: 0000000000000006
[11402.968884] RDX: ffffffff0c930020 RSI: ff3eefdafcacfd70 RDI: ff2d1acd00409140
[11402.971167] RBP: ff3eefdafcacfda4 R08: 8080808080808080 R09: fefefefefefefeff
[11402.973512] R10: 0000303539343037 R11: 0000000000000004 R12: ff2d1acd8cb0b7e0
[11402.975915] R13: ff3eefdafcacfd70 R14: 0000000000000000 R15: 0000000000000000
[11402.978356] FS: 00007f757eaed640(0000) GS:ff2d1b89b5a42000(0000) knlGS:0000000000000000
[11402.980842] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[11402.983374] CR2: ffffffff0c93001c CR3: 00000004ba18e002 CR4: 0000000000771ef0
[11402.986004] PKRU: 55555554
[11402.988640] Call Trace:
[11402.991231] <TASK>
[11402.993831] lookup_fast+0x2b/0x100
[11402.996468] walk_component+0x1f/0x150
[11402.999080] link_path_walk+0x105/0x2a0
[11403.001714] path_openat+0x99/0x2b0
[11403.004319] ? srso_alias_return_thunk+0x5/0xfbef5
[11403.006983] ? up_read+0x5a/0x90
[11403.009610] ? srso_alias_return_thunk+0x5/0xfbef5
[11403.012250] ? do_user_addr_fault+0x190/0x6a0
[11403.014927] do_filp_open+0xd3/0x180
[11403.017635] ? __pfx_kfree_link+0x10/0x10
[11403.020383] ? srso_alias_return_thunk+0x5/0xfbef5
[11403.023097] do_sys_openat2+0x8a/0xe0
[11403.025765] __x64_sys_openat+0x69/0xa0
[11403.028424] do_syscall_64+0x64/0xba0
[11403.031066] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[11403.033733] RIP: 0033:0x7f75816ff1e4
[11403.036336] Code: 24 20 eb 8f 66 90 44 89 54 24 0c e8 26 8c f8 ff 44 8b 54 24 0c 44 89 e2 48 89 ee 41 89 c0 bf 9c ff ff ff b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 77 34 44 89 c7 89 44 24 0c e8 78 8c f8 ff 8b 44
[11403.044367] RSP: 002b:00007f757eaeb870 EFLAGS: 00000293 ORIG_RAX: 0000000000000101
[11403.047072] RAX: ffffffffffffffda RBX: 00007f757eaed5e0 RCX: 00007f75816ff1e4
[11403.049795] RDX: 0000000000080000 RSI: 00007f757edd90e2 RDI: 00000000ffffff9c
[11403.052470] RBP: 00007f757edd90e2 R08: 0000000000000000 R09: 0000000000000000
[11403.055070] R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000080000
[11403.057594] R13: 00007f758187fe50 R14: 00007f75818e5040 R15: 0000000000000001
[11403.060096] </TASK>
[11403.062521] Modules linked in: vhost_net(E) vhost(E) vhost_iotlb(E) tap(E) tun(E) ceph(E) libceph(E) cts(E) krb5enc(E) authenc(E) camellia_aesni_avx2(E) camellia_aesni_avx_x86_64(E) camellia_x86_64(E) camellia_generic(E) cmac(E) krb5(E) dns_resolver(E) netfs(E) nf_tables(E) tcp_diag(E) udp_diag(E) inet_diag(E) netconsole(E) sctp(E) ip6_udp_tunnel(E) udp_tunnel(E) rfkill(E) nfnetlink_cttimeout(E) nfnetlink(E) openvswitch(E) nf_conncount(E) nf_nat(E) nsh(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) sunrpc(E) kvm_amd(E) sr_mod(E) kvm(E) irqbypass(E) amd_atl(E) cdrom(E) sd_mod(E) amd64_edac(E) wmi_bmof(E) cdc_ether(E) acpi_ipmi(E) sg(E) edac_mce_amd(E) acpi_cpufreq(E) usbnet(E) pcspkr(E) ipmi_si(E) mii(E) ipmi_ssif(E) i2c_piix4(E) ipmi_devintf(E) vfat(E) i2c_smbus(E) k10temp(E) ipmi_msghandler(E) wmi(E) fat(E) i2c_core(E) sch_fq_codel(E) fuse(E) ext4(E) crc16(E) mbcache(E) jbd2(E) raid1(E) usb_storage(E) nvme(E) ahci(E) libahci(E) mpt3sas(E) raid_class(E) xhci_pci(E) nvme_core(E) ghash_clmulni_intel(E) libata(E)
[11403.062611] scsi_transport_sas(E) i40e(E) nvme_keyring(E) libie(E) nvme_auth(E) xhci_hcd(E) sp5100_tco(E) libie_adminq(E) hkdf(E) dm_mod(E) dax(E) aesni_intel(E)
[11403.100478] CR2: ffffffff0c93001c
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette