Re: [PATCH printk v2] printk: Postpone kthread startup

[Petr Mladek]

On Fri 2026-10-16 09:36:41, Derek Barbosa wrote:
> On Fri, Oct 16, 2026 at 11:00:34AM +0206, John Ogness wrote:
> > There have been various bug reports related to SMP systems hanging
> > on boot. The reports vary but seem to have some overlap with regard
> > to missed IPI's during SMP setup. The reports also show that
> > reverting NBCON console usage allows the bugs to disappear.
> > 
> > The NBCON kthreads for console printing are started very early
> > during boot using early_initcall(). Since these kthreads are
> > asynchronously enabling/disabling interrupts during early boot, they
> > may be interfering with early SMP setup on some hardware platforms.
> > 
> > Starting the console printing kthreads later has shown to avoid the
> > hanging on systems that could previously reproduce the issue.
> > 
> > Postpone kthread starting until after all arch initcalls have
> > completed.
> > 
> > Reported-by: Derek Barbosa <debarbos@xxxxxxxxxx>
> > Closes: https://lore.kernel.org/linux-rt-users/arPmYPbFLf2EldIl@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> > Fixes: 76f258bf3f2a ("printk: nbcon: Introduce printer kthreads")
> > Signed-off-by: John Ogness <john.ogness@xxxxxxxxxxxxx>
> > ---
> >  v1: https://lore.kernel.org/lkml/87ecdrqfhw.fsf@xxxxxxxxxxxxxxxxxxxxx
> > 
> >  changes since v1:
> >  - Reword commit message.
> >  - Rebase from 6.6-rt to 7.3-rc.
> > 
> >  kernel/printk/printk.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
> > index 0ed86b5e8b160..9a205f94bfac7 100644
> > --- a/kernel/printk/printk.c
> > +++ b/kernel/printk/printk.c
> > @@ -3864,7 +3864,7 @@ static int __init printk_set_kthreads_ready(void)
> >  
> >  	return 0;
> >  }
> > -early_initcall(printk_set_kthreads_ready);
> > +subsys_initcall(printk_set_kthreads_ready);
> >  #endif /* CONFIG_PRINTK */
> >  
> >  static int __read_mostly keep_bootcon;
> > 
> > base-commit: 38dfd607b58b63b5488d5756786ee4ee499eebb8
> > -- 
> > 2.47.3
> > 
> 
> In the v1 thread, I mentioned that I would leave the reproducer spinning in the
> test environment for a few more hours. It has been well over 9 hours. No signs
> of the existing issue popping up.
> 
> Tested-by: Derek Barbosa <debarbos@xxxxxxxxxx>

Thanks a lot for testing this. It is good to know that it helped.

I am a bit nervous that we do not know what is the culprit.
For example, I wonder if the same problem might happen during
CPU hot-plug at runtime.

Anyway, I had a brain storming session with John and we were not able
to find any hole in the nbcon API design. We did not find any location
where it could cause deadlock. All busy loops are under
con->device_lock(), aka port->lock. And the port lock is taken with
IRQs disabled. And neither the con->device_lock() nor the busy loop
is done in NMI.

So, this rather looks like an existing problem. And the parallel
handling of the serial console just made it more likely to happen.

The converted uart 8250 is being used on many production systems
which are running the RT kernel and got this change from the external
RT patchset. And I have heard about this problem for the first time just
a few weeks ago. This makes me believe that it affects only small
amount of HW.

This change clearly helped. Which makes me even more confident
that we could keep the uart 8250 driver converted in 7.3.
I am going to commit this patch and create a pull request...

Best Regards,
Petr