Re: [PATCH v2] stop_machine: Defer legacy console flushes while a CPU runs a stopper callback

[Petr Mladek]

On Fri 2026-09-11 16:32:40, Aditya Chillara wrote:
> On 9/10/2026 9:46 PM, Bradley Morgan wrote:
> > On 10 September 2026 04:53:55 BST, Aditya Chillara
> > <aditya.chillara@xxxxxxxxxxxxxxxx> wrote:
> >> The cpu stopper thread runs above every other scheduling class, so while a
> >> stopper callback executes, nothing else on that CPU is scheduled. If such
> >> a
> >> callback emits a normal-priority printk(), the legacy console path can
> >> synchronously flush the pending console backlog. On systems with a slow
> >> UART and a large backlog, this holds the CPU long enough to starve RT
> >> kthreads such as the watchdog pet, and for multi_cpu_stop() prevents the

Which watchdog, please?
Please, include the entire watchdog report into the commit message.

Also please mention the real life scenario (with SELinux logging)
in the commit message if there is a real life scenario, see below.

> >> CPU from advancing the state machine while the other CPUs wait.

[...]

> >> stop_machine() (used here for kprobe text patching) stops every CPU by
> >> running multi_cpu_stop() on each of them, through the per-CPU
> >> "migration/%u" threads. These threads run at a higher priority than the
> >> msm_watchdog thread. At bite time, all eight CPUs were still spinning in
> >> multi_cpu_stop()'s MULTI_STOP_PREPARE state, where interrupts are left
> >> enabled.
> >>
> >> Heavy SELinux denial logging had built up a large backlog on the
> >> console.

Did this happen in production, please?

> >> Reproduced and verified with an out-of-tree test module that triggers
> >> stop_machine() with a queued console backlog and a printk() inside the
> >> rendezvous, paired with a kprobe-based script that flags any console
> >> flush happening while a CPU is inside a stopper callback.

I ask because the reproducer is artificial and the solution is just
a band aid. printk_deferred_enter() and printk_deferred() is
a whack-a-mole solution.

They were created to avoid deadlocks on the run queue lock. It might
happen when printk() is called in the scheduler code and the legacy
console_unlock() tries to wakeup a waiting process.

In addition, printk_deferred_enter() is used on few locations when
many debug messages are printed in an interrupt context.

AFAIK, this would be the first case when printk() deferred is used
to avoid softlockup in a code when the many messages are printed
from another context. And I am not sure if we want to go this way.

The legacy consoles might trigger a softlockup when printk() is called
with IRQs disabled and it has to flush many pending messages.

I do not see anything special about multi_cpu_stop() except that
all CPUs are bloked in this case. OK, blocking all CPUs is
much worse than blocking one CPU. And it might be reason
to accept this patch.

But you might be able to create an artificial reproducer which might
trigger softlockup by any printk() called with IRQs disabled. And
we do not want to convert all printk() calls to printk_deferred(),
definitely. We need to define a boundary.

Note that the preferred solution is to:

1. Convert console drivers to NBCON API which would defer most
   printk() calls out of box. There are only few selected
   emergency situations when printk() tries to flush even
   nbcon consoles immediately. But even the emergency flush
   is bound against parallel writers.

2. Reduce loglevel so that only the most critical messages
   are flushed when slow consoles are used.

Note that there is pending also a patchset which would allow to
set per-console loglevel.

Best Regards,
Petr