[BUG] arm64/sched: hard lockup with rq->__lock stuck locked and no apparent owner

[含笑傲月]

Hi,


We encountered a hard lockup on an ARM64 server where a scheduler
runqueue qspinlock remained locked while essentially all active CPUs
were waiting for the same lock.


We have a vmcore and have done some initial analysis. At this point we
cannot determine whether this is a scheduler/qspinlock issue, memory
corruption, or a platform/CPU coherency issue. We would appreciate help
reviewing the state captured in the vmcore.


The most unusual observation is that the qspinlock queue appears
internally consistent, but we cannot identify a CPU which is still
executing as the owner of the lock.


## System information


Architecture:


```text
arm64
```


Platform:


```text
Huawei/HiSilicon Kunpeng platform
Great Wall QingTian RK5260 V5
```


BIOS:


```text
Huawei Corp. 6.65
2023-09-20
```


Kernel:


```text
5.15.131-5.xxxx.aarch64
```


Kernel build:


```text
#1 SMP Thu Feb 22 16:01:25 CST 2024
```


CPU count:


```text
128
```


Memory:


```text
512 GB
```


Uptime before the failure:


```text
744 days, 09:09:29
```


Panic:


```text
Kernel panic - not syncing: Hard LOCKUP
```


The supplied vmlinux has been verified against the vmcore kallsyms.


The dump is a partial kdump, but an SDEI watchdog record preserved
interrupted register state for all 128 CPUs.


## Observed runqueue lock state


The affected lock is:


```text
&runqueues[3].__lock = ffff003ff6e3b540
```


The lock value in the vmcore is:


```text
counter = 0x00d00101
locked  = 1
pending = 1
tail    = 0x00d0
```


In other words, the qspinlock still indicates an active owner, a
pending waiter, and an MCS queue.


`rq[3]` state includes:


```text
rq->cpu        = 3
rq->nr_running = 3
rq->curr       = ffff0020d2770600
```


## CPU3 state


The SDEI interrupted register state for CPU3 is:


```text
pc  = queued_spin_lock_slowpath+0x200
x3  = ffff003ff6e3b540
x19 = ffff003ff6e3b540
x0  = ffff003ff6e3c248
```


Here `x3` and `x19` point to `rq[3].__lock`.


`x0` points into CPU3's per-CPU qnode rather than to the runqueue lock,
which is consistent with CPU3 already having joined the qspinlock MCS
wait queue.


The reconstructed CPU3 kernel stack is approximately:


```text
queued_spin_lock_slowpath
do_raw_spin_lock
raw_spin_rq_lock_nested
__schedule
schedule
futex_wait_queue_me
futex_wait
do_futex
...
```


The task is:


```text
PID: 648437
COMMAND: ovs-monitor
```


Therefore CPU3 appears to be entering `schedule()` through a normal
futex wait path and is blocked while trying to acquire its runqueue
lock.


## qspinlock waiter state


We also inspected the per-CPU qnodes.


The MCS queue can be reconstructed as a valid linked list. In the
captured state:


- 124 CPUs have active MCS qnodes for this lock
- CPU116 is on the pending path
- CPU29 is at the MCS queue head
- CPU51 is at the MCS queue tail


The encoded qspinlock tail is:


```text
0xd0
```


Using the normal qspinlock tail encoding:


```text
tail CPU = (0xd0 >> 2) - 1 = 51
tail idx = 0
```


This agrees with the reconstructed MCS queue, whose last node is the
CPU51 qnode.


This consistency is one reason we currently do not think the lock word
was simply overwritten with an arbitrary value.


The queue structure, qnode state, and encoded qspinlock tail all appear
mutually consistent.


The unusual part is the locked bit.


## No obvious owner


At the time of the watchdog snapshot, CPUs waiting on this runqueue
lock are stopped in `queued_spin_lock_slowpath()`.


The remaining CPUs which are not part of the waiter chain appear to be
in idle paths.


So far we have not found a CPU executing in a scheduler critical
section which can explain:


```text
rq[3].__lock.locked == 1
```


From the captured CPU execution state we currently see:


```text
                     rq[3].__lock
                       locked=1
                          |
                no apparent active owner
                          |
                     pending waiter
                          |
                     MCS queue head
                          |
                         ...
                          |
                     MCS queue tail
```


The MCS wait queue itself appears healthy. Progress stops because its
head never observes the locked/pending portion becoming available.


CPU29, which is at or near the queue head, is spinning in the part of
`queued_spin_lock_slowpath()` which repeatedly loads the qspinlock and
waits for the low 16 bits to clear.


## Possible recursive owner/waiter case


We specifically considered whether one CPU could have acquired the
runqueue lock and then recursively attempted to acquire the same lock,
thereby becoming both the owner and a waiter.


CPU3 is definitely a waiter for `rq[3].__lock`.


However, its captured call chain is:


```text
futex wait
  -> schedule()
     -> __schedule()
        -> raw_spin_rq_lock_nested()
           -> queued_spin_lock_slowpath()
```


We currently have no evidence that CPU3 had already acquired this same
rq lock before entering `schedule()`.


Also, `rq[3].curr` being the CPU3 task does not by itself imply that
CPU3 owns `rq[3].__lock`. CPU3 has not completed the context switch
because it is blocked acquiring the lock.


We therefore cannot currently demonstrate a recursive runqueue-lock
deadlock.


## Why we are reporting this


The state looks different from a typical lock-ordering deadlock:


1. The qspinlock remains locked.
2. The MCS queue is structurally consistent.
3. The encoded qspinlock tail agrees with the actual per-CPU qnode chain.
4. A very large number of CPUs have accumulated as waiters.
5. We have not yet identified a corresponding active lock owner.


One possible explanation would be that the original owner logically
left the critical section, but the transition of the qspinlock locked
byte from `1` to `0` did not become observable as expected.


On arm64, qspinlock unlock eventually releases the locked byte with a
release store. We are investigating whether the captured state could
result from a software race around that operation, or whether a
platform-level store/coherency problem must also be considered.


We are not claiming at this point that this is a CPU hardware bug.


Other possibilities such as scheduler misuse, recursive locking,
qspinlock state corruption, memory corruption, or an issue in our
vendor kernel still need to be ruled out.


## Questions


We would particularly appreciate comments on the following:


1. Is there any valid scheduler/qspinlock sequence where `rq->__lock`
   can remain with:


   ```text
   locked = 1
   pending = 1
   valid MCS queue
   ```


   while the original lock holder is no longer visible in any CPU's
   execution context?


2. Could a CPU which originally owned `rq->__lock` legitimately enter
   `queued_spin_lock_slowpath()` for the same lock through
   `__schedule()`, without being caught as an atomic scheduling or
   recursive locking bug?


3. Are there known qspinlock races in older 5.15 arm64 kernels which
   could leave an orphaned locked bit while preserving a valid MCS
   queue?


4. Are there additional qspinlock, scheduler, or arm64 state fields we
   should extract from this vmcore to conclusively identify the last
   owner?


5. In particular, is there a way to distinguish from the dump alone
   between:


   ```text
   a) recursive owner -> waiter deadlock
   b) software corruption of the lock state
   c) a missed or stale unlock observation
   ```


## Additional information


We still have:


- the vmcore
- matching vmlinux
- saved SDEI register state for all CPUs
- per-CPU qnode state
- per-CPU runqueue state
- disassembly of `queued_spin_lock_slowpath()`
- scheduler stacks
- detailed reconstruction of the MCS queue


We can provide additional `crash(8)` output or inspect specific fields
if that would help.


Unfortunately this failure is extremely rare. The affected machine had
been running for approximately 744 days when the hard lockup occurred,
and we do not yet have a reliable reproducer.


Thanks for any suggestions on how to narrow this down.


Regards,  
Liu Wei