Re: [PATCH v4] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

[Aaron Tomlin]

On Thu, Oct 22, 2026 at 09:20:59AM -0400, Aaron Tomlin wrote:
> On Thu, Oct 22, 2026 at 02:36:55PM +0200, Frederic Weisbecker wrote:
> > (Adding Marco)
> > 
> > Le Wed, Oct 21, 2026 at 11:47:37PM -0400, Aaron Tomlin a écrit :
> > > Commit a46c27026da1 ("blk-mq: don't schedule block kworker on isolated
> > > CPUs") excluded isolated CPUs from hctx->cpumask to prevent worker
> > > threads from executing block work on isolated CPUs. When all CPUs mapped
> > > to a hardware queue are isolated (i.e. blk_mq_hctx_empty_cpumask(hctx)
> > > evaluates to true), blk_mq_hctx_next_cpu() returns WORK_CPU_UNBOUND with
> > > the intention of offloading execution.
> > > 
> > > However, kblockd_workqueue is allocated as a per-CPU workqueue
> > > (WQ_PERCPU). When WORK_CPU_UNBOUND is queued on a per-CPU workqueue, the
> > > workqueue core resolves the target CPU via raw_smp_processor_id().
> > > Consequently, if a task executing on an isolated CPU submits I/O and
> > > triggers an asynchronous queue dispatch with zero delay (e.g. upon
> > > hardware queue congestion, tag/resource exhaustion returning
> > > BLK_STS_RESOURCE, or plugging), &hctx->run_work is queued directly onto
> > > the local isolated CPU's worker pool. This causes the local per-CPU
> > > high-priority kworker (e.g. "kworker/6:1H-kblockd") to be scheduled on
> > > the isolated CPU. In real-time environments, an active SCHED_FIFO workload
> > > will starve this SCHED_NORMAL worker, causing hardware dispatch to stall,
> > > in addition to introducing unwanted latency jitter on nohz_full cores,
> > > thereby defeating the original objective of commit a46c27026da1.
> > > 
> > > Introduce blk_mq_hctx_unbound_cpu() to select an online housekeeping CPU
> > > from housekeeping_cpumask(HK_TYPE_DOMAIN) via
> > > cpumask_any_and_distribute() when executing on an isolated CPU, and
> > > otherwise fall back to WORK_CPU_UNBOUND. Use this helper across the
> > > unbound fallback paths in blk_mq_hctx_next_cpu().
> > > 
> > > Additionally, for single-queue devices, gate the check behind the static
> > > key housekeeping_overridden under CONFIG_CPU_ISOLATION to ensure that
> > > common configurations without CPU isolation incur zero overhead on the
> > > fast path.
> > > 
> > > This ensures that:
> > >     1.  When executing on an isolated CPU, deferred queue work is
> > >         cleanly offloaded to an online housekeeping CPU across single-queue
> > >         devices, hardware queues with empty cpumasks, and offline CPU
> > >         fallback paths.
> > > 
> > >     2.  The per-CPU high-priority kworker on the isolated CPU is not
> > >         scheduled, avoiding priority inversion and latency jitter.
> > > 
> > >     3.  When executing on a non-isolated CPU, WORK_CPU_UNBOUND is
> > >         preserved, maintaining optimal local execution with zero CPU
> > >         bouncing while avoiding timer pinning hazards.
> > > 
> > >     4.  On systems without CPU isolation, single-queue devices maintain
> > >         the zero-overhead fast path via a static key.
> > > 
> > > Fixes: a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs")
> > > Cc: Ming Lei <ming.lei@xxxxxxxxxx>
> > > Cc: Jens Axboe <axboe@xxxxxxxxx>
> > > Cc: stable@xxxxxxxxxxxxxxx
> > > Signed-off-by: Aaron Tomlin <atomlin@xxxxxxxxxxx>
> > > ---
> > > Changes since v3:
> > > 
> > >  - Create companion blktests test case (block/050) to reproduce this issue
> > >    and verify the fix (Daniel Wagner)
> > > 
> > >  - Gate the single-queue (nr_hw_queues == 1) check behind the static key
> > >    housekeeping_overridden under CONFIG_CPU_ISOLATION to ensure the
> > >    common case without CPU isolation incurs zero overhead on the fast
> > >    path (Jens Axboe)
> > > 
> > >  - Wrap the evaluation of cpu_is_isolated() inside guard(rcu)() in
> > >    blk_mq_hctx_unbound_cpu() to prevent an RCU lockdep warning and ensure
> > >    safe dereferencing of the housekeeping cpumask while concurrent cpuset
> > >    updates occur
> > > 
> > >  - Disable preemption via guard(preempt)() around cpumask_any_and_distribute()
> > >    to prevent a preemption warning under CONFIG_PREEMPT_RCU and avoid
> > >    data races when accessing per-CPU distribution state
> > > 
> > >  - Link to v3: https://lore.kernel.org/all/20261016052203.489760-1-atomlin@xxxxxxxxxxx/
> > > 
> > > Changes since v2:
> > > 
> > >  - Query housekeeping_cpumask(HK_TYPE_DOMAIN) instead of HK_TYPE_TIMER
> > >    to align with cpu_is_isolated(), workqueue affinity rules, and cpuset
> > >    isolated partitions
> > > 
> > >  - Query the housekeeping cpumask under guard(rcu)() to safely handle
> > >    dynamic cpuset updates without lockdep warnings
> > > 
> > >  - Validate that cpumask_any_and_distribute() returns a valid cpu ID
> > >    before returning, falling back cleanly to WORK_CPU_UNBOUND
> > > 
> > >  - Link to v2: https://lore.kernel.org/lkml/20261015020225.422107-1-atomlin@xxxxxxxxxxx/
> > > 
> > > Changes since v1:
> > > 
> > >  - Introduce the blk_mq_hctx_unbound_cpu() helper to steer deferred work
> > >    to an online housekeeping CPU only when executing on an isolated
> > >    core, preserving WORK_CPU_UNBOUND and local cache locality for
> > >    housekeeping cores
> > > 
> > >  - Apply blk_mq_hctx_unbound_cpu() across all unbound fallback paths in
> > >    blk_mq_hctx_next_cpu(), comprehensively covering single-queue devices
> > >    (nr_hw_queues == 1), empty hardware queue cpumasks, and the offline
> > >    CPU fallback path.
> > > 
> > >  - Elaborate in the commit log on the triggers for asynchronous queue
> > >    dispatch (e.g. tag exhaustion, BLK_STS_RESOURCE backpressure, and
> > >    plugging) and the SCHED_FIFO worker starvation hazard on isolated
> > >    cores
> > > 
> > >  - Link to v1: https://lore.kernel.org/lkml/20261012012905.267920-1-atomlin@xxxxxxxxxxx/
> > > ---
> > >  block/blk-mq.c | 31 ++++++++++++++++++++++++++++---
> > >  1 file changed, 28 insertions(+), 3 deletions(-)
> > > 
> > > diff --git a/block/blk-mq.c b/block/blk-mq.c
> > > index 025a799f35b9..f8f862a0e525 100644
> > > --- a/block/blk-mq.c
> > > +++ b/block/blk-mq.c
> > > @@ -2241,6 +2241,23 @@ static bool blk_mq_hctx_empty_cpumask(struct blk_mq_hw_ctx *hctx)
> > >          return hctx->next_cpu >= nr_cpu_ids;
> > >  }
> > >  
> > > +static inline int blk_mq_hctx_unbound_cpu(void)
> > > +{
> > > +	guard(rcu)();
> > > +
> > > +	if (cpu_is_isolated(raw_smp_processor_id())) {
> > > +		int cpu;
> > > +
> > > +		guard(preempt)();
> > > +		cpu = cpumask_any_and_distribute(housekeeping_cpumask(HK_TYPE_DOMAIN),
> > > +						 cpu_online_mask);
> > > +		if (cpu < nr_cpu_ids)
> > > +			return cpu;
> > > +	}
> > > +
> > > +	return WORK_CPU_UNBOUND;
> > > +}
> > 
> > Just for future record, we are working on a WQ_PREFER_PERCPU workqueue flag that
> > should automatically handle this sort of things. It's not ready for now so these
> > hacks are still necessary but keep in mind this is a good candidare for future
> > simplication.
> > 
> > Thanks.
> > 
> > -- 
> > Frederic Weisbecker
> > SUSE Labs
> 
> Hi Frederic,
> 
> Noted and thank you for sharing.
> 
> Sean and I observed this behaviour on a kernel with commit a46c27026da1
> ("blk-mq: don't schedule block kworker on isolated CPUs") applied.
> 
> We will certainly keep WQ_PREFER_PERCPU in mind as a cleaner long-term
> path.

Hi Jens,

Would this patch be suitable for inclusion while we await the introduction
of WQ_PREFER_PERCPU?

Kind regards,
-- 
Aaron Tomlin