Re: [RFC 1/1] sched/rt: Avoid RTO push to stopped CPUs during shutdown

[Peter Zijlstra]

On Tue, Oct 13, 2026 at 11:01:40AM +0800, wangqk1 wrote:
> During system poweroff, machine_power_off() invokes smp_send_stop(). A
> CPU receiving IPI_CPU_STOP parks without running the CPU hotplug teardown.
> Its RT overload state can therefore retain a stale rto_mask bit.
> 
> A CPU still executing the RTO irq_work iterator can select that stale bit
> and queue work to the stopped CPU.
> 
> Observed call trace:
>   irq_work_queue_on()
>   rto_push_irq_work_func()
>   irq_work_single()
>   __flush_smp_call_function_queue()
>   generic_smp_call_function_single_interrupt()
>   ipi_handler()
> 
> Stop RTO forwarding after terminal system states begin and skip rto_mask
> candidates that are already offline.
> 
> This is a workaround, not a complete synchronization protocol. The
> cpu_online() test is not atomic with irq_work_queue_on(), and panic or
> crash-kexec paths do not set a terminal system_state before stopping CPUs.
> 
> Signed-off-by: wangqk1 <wangqk1@xxxxxxxxxxxx>
> ---
>  kernel/sched/rt.c | 20 +++++++++++++++++++-
>  1 file changed, 19 insertions(+), 1 deletion(-)
> 
> diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
> index 85303add726d..716ac30c5afb 100644
> --- a/kernel/sched/rt.c
> +++ b/kernel/sched/rt.c
> @@ -2084,6 +2084,14 @@ static void push_rt_tasks(struct rq *rq)
>  
>  #ifdef HAVE_RT_PUSH_IPI
>  
> +static inline bool rto_push_allowed(void)
> +{
> +	enum system_states state = READ_ONCE(system_state);
> +
> +	return state != SYSTEM_HALT && state != SYSTEM_POWER_OFF &&
> +	       state != SYSTEM_RESTART;
> +}
> +
>  /*
>   * When a high priority task schedules out from a CPU and a lower priority
>   * task is scheduled in, a check is made to see if there's any RT tasks
> @@ -2131,6 +2139,11 @@ static int rto_next_cpu(struct root_domain *rd)
>  	int next;
>  	int cpu;
>  
> +	if (unlikely(!rto_push_allowed())) {
> +		rd->rto_cpu = -1;
> +		return -1;
> +	}
> +
>  	/*
>  	 * When starting the IPI RT pushing, the rto_cpu is set to -1,
>  	 * rt_next_cpu() will simply return the first CPU found in
> @@ -2155,8 +2168,13 @@ static int rto_next_cpu(struct root_domain *rd)
>  		if (cpu == this_cpu)
>  			continue;
>  
> -		if (cpu < nr_cpu_ids)
> +		if (cpu < nr_cpu_ids) {
> +			/* smp_send_stop() does not clear stale rto_mask bits. */
> +			if (!cpu_online(cpu))
> +				continue;
> +
>  			return cpu;
> +		}
>  
>  		rd->rto_cpu = -1;

This is horrific; lets not do this. The bug really is at the place where
you randomly and without warning take CPUs down. We're not adding random
hacks for things like that.