Re: [PATCH v16 0/9] blk: honor isolcpus configuration

[Ionut Nechita]

Hi Aaron, Daniel,

As promised, here are the end-to-end results from exercising v16 on our
6.18 LTS PREEMPT_RT tree. Short version: managed_irq_strict behaves
exactly as documented on multiqueue NVMe with a deliberately narrow
housekeeping pool, including the nr_vectors >= nr_cpus case.

Series under test (as backported to 6.18 LTS)
---------------------------------------------
  0001 scsi-aacraid: use block-layer helpers for queue count
  0002 lib/group_cpus: remove dead SMP code
  0003 lib/group_cpus: add group_mask_cpus_evenly()
  0004 sched/isolation: prevent out-of-bounds read in isolcpus parsing
  0005 isolation: introduce the managed_irq_strict isolcpus type
  0006 blk-mq: use housekeeping CPUs only with isolcpus=managed_irq_strict
  0007 blk-mq: prevent offlining housekeeping CPUs with online-associated
       queues
  0008 genirq/affinity: restrict managed IRQ affinity to housekeeping CPUs
  0009 docs: document the managed_irq_strict isolcpus flag
  0010 genirq/affinity: confine reserved pre/post vectors to housekeeping
       CPUs

Note: 0010 is a local follow-up (see section 3), not part of the
upstream v16 series.

Test environment
----------------
  Kernel:   6.18.15-rt and 6.18.52-rt (PREEMPT_RT), v16 backported
            (6.18.52 is the current 6.18 LTS; behaviour is identical on
            both, the data below is from 6.18.15)
  CPU:      single-socket Intel Xeon (Granite Rapids), 40C/80T,
            1 NUMA node (all 80 CPUs on node0)
  Storage:  2x NVMe (SK hynix PE9010), b6:00.0 and b7:00.0
  NICs:     8-port E830-CC + 4-port E825-C (managed IRQs)
  MSI-X:    2235 vectors allocated system-wide

  cmdline (relevant bits):
    nohz_full=2-38,42-78
    isolcpus=nohz,domain,managed_irq_strict,2-38,42-78
    rcu_nocbs=2-39,42-79 kthread_cpus=0-1,40-41 irqaffinity=39,79

  => isolated (isolcpus) = 2-38,42-78   (74 CPUs)
     housekeeping (HK)   = 0,1,39,40,41,79   (6 CPUs)

Because HK is only 6 CPUs while the box carries 2235 vectors, this is
squarely the "narrow housekeeping pool, nr_vectors >= nr_cpus" scenario.

1) blk-mq queue allocation and mapping (patches 6, 7)
-----------------------------------------------------
Both NVMe devices allocate exactly 6 hardware queues, matching the HK
weight. Each hctx is owned by exactly one HK CPU:

    nvme0n1 / nvme1n1:
      hctx0: 0   hctx1: 1   hctx2: 39
      hctx3: 40  hctx4: 41  hctx5: 79

No isolated CPU owns a queue. The sysfs cpu_list only shows the HK CPUs
because blk_mq_map_swqueue() strips the isolated CPUs from hctx->cpumask
(as intended). The full software-context routing from debugfs confirms
the 74 isolated CPUs are still mapped onto these 6 HK queues rather than
left unserved, evenly (~12-13 isolated CPUs per queue), so I/O submitted
from an isolated core completes on a housekeeping CPU:

    hctx0 (owner 0):  0,2,8,14,20,26,32,38,47,53,59,65,71,77
    hctx1 (owner 1):  1,3,9,15,21,27,33,42,48,54,60,66,72,78
    hctx2 (owner 39): 4,10,16,22,28,34,39,43,49,55,61,67,73
    hctx3 (owner 40): 5,11,17,23,29,35,40,44,50,56,62,68,74
    hctx4 (owner 41): 6,12,18,24,30,36,41,45,51,57,63,69,75
    hctx5 (owner 79): 7,13,19,25,31,37,46,52,58,64,70,76,79

All 80 CPUs are covered exactly once. This also lines up with the
hotplug guard below: isolated CPU 3 is routed to hctx1 (owned by HK CPU
1), which is precisely why offlining CPU 1 is refused.

2) Managed IRQ affinity (patch 8)
---------------------------------
Every NVMe completion-queue vector has its effective_affinity confined
to the HK set:

    nvme0q1..q6 eff = 0, 1, 39, 40, 41, 79
    nvme1q1..q6 eff = 0, 1, 39, 40, 41, 79

No managed interrupt targets an isolated CPU.

3) Reserved pre/post vectors
----------------------------
The NVMe admin (pre_vector) queues also stay off the isolated set:

    nvme0q0 eff = 79
    nvme1q0 eff = 39

Note: in the stock series the reserved pre/post vectors still default to
irq_default_affinity, which on this config spans the isolated CPUs, so a
freshly probed admin queue can briefly land on an isolated core before
any userspace re-steer. On PREEMPT_RT that stray wakeup is precisely
what we are trying to avoid, so we carry a small follow-up that confines
the affd->pre_vectors / post_vectors to the housekeeping mask when
HK_TYPE_MANAGED_IRQ_STRICT is set. The affinities above are with that
follow-up applied. Happy to post it separately if there is interest in
closing that gap upstream.

4) blk-mq hotplug guard (patch 7)
---------------------------------
Offlining a housekeeping CPU that still serves an online isolated CPU is
refused:

    # echo 0 > /sys/devices/system/cpu/cpu1/online
    -> fails (-EINVAL)
    blk-mq: cannot offline CPU 1: online isolated CPU 3 is still
            mapped to hctx1

After offlining the dependent isolated CPUs first, offlining the HK CPU
succeeds. suspend/hibernate (cpuhp_tasks_frozen) is correctly exempted.

5) Vector count / probe robustness (patch 8)
--------------------------------------------
With 2235 vectors against 6 HK CPUs, no device aborted probe: no -ENOSPC
and no nr_io_queues=0. The minimum-vector safety net in
irq_calc_affinity_vectors() holds on this narrow-HK box.

Backport notes (6.18.15 / 6.18.52)
----------------------------------
Nothing behavioural, just tree deltas: the default isolcpus flag stays
HK_FLAG_DOMAIN (no HK_FLAG_DOMAIN_BOOT in this tree); the cpus_read_lock
around the cpu_possible_mask weight in irq_calc_affinity_vectors() is
retained; and the housekeeping.rst hunk from the docs patch is dropped
(that guide does not exist here).

6) Runtime completion locality (PREEMPT_RT threaded IRQs)
---------------------------------------------------------
fio randread 64k, 4 jobs, iodepth=64, 120s, pinned to a single isolated
CPU via a device-plugin extended resource on Kubernetes.

  Results:
    IOPS       : 56.9k
    Bandwidth  : 3559 MiB/s (3732 MB/s), 417 GiB in 120s
    clat       : avg 4489us, stdev 30.99us, p99 4555us
    total IOs  : 6,833,178

While fio was submitting from that isolated CPU, every threaded NVMe
IRQ handler on the host was scheduled on a housekeeping CPU
(SCHED_FIFO, as expected on PREEMPT_RT). `ps -eLo psr,cls,comm |
grep irq.*nvme` during the run:

   0 FF irq/195-nvme0q3   0 FF irq/196-nvme0q4   0 FF irq/198-nvme0q6
   1 FF irq/187-nvme1q2   1 FF irq/188-nvme1q3   1 FF irq/194-nvme0q2
  39 FF irq/185-nvme1q0
  40 FF irq/186-nvme1q1  40 FF irq/189-nvme1q4  40 FF irq/193-nvme1q6
  41 FF irq/191-nvme1q5  41 FF irq/192-nvme0q1  41 FF irq/197-nvme0q5
  79 FF irq/190-nvme0q0

All 14 threaded handlers land on {0,1,39,40,41,79} = HK. Zero
irq/*-nvme* threads scheduled on any of the 74 isolated CPUs.

The /proc/interrupts delta over the 120s run confirms it at the counter
level. Every nvme* IRQ that fired incremented only on an HK CPU:

  irq 192 nvme0q1 CPU0 : +6,809,271   (fio's IOs, one HK owner)
  irq 187 nvme1q2 CPU1 : +684
  irq 186 nvme1q1 CPU0 : +598
  irq 189 nvme1q4 CPU40: +568
  irq 191 nvme1q5 CPU41: +512
  irq 188 nvme1q3 CPU39: +304
  irq 193 nvme1q6 CPU79: +271
  irq 196 nvme0q4 CPU40: +4
  irq 194 nvme0q2 CPU1 : +3

Every single delta is on {0,1,39,40,41,79}. Zero completions on any of
the 74 isolated CPUs. I/O submitted by a task pinned to an isolated
core completes entirely on housekeeping cores.

Still on my list: a negative-control boot with plain managed_irq to
show the confinement disappears. I will follow up with that.

On option 1 vs option 2, my earlier preference stands: keep
managed_irq_strict as an explicit opt-in and leave managed_irq
semantics unchanged, to avoid a silent behavioural change for existing
deployments.

Tested-by: Ionut Nechita <ionut.nechita@xxxxxxxxxxxxx>

Thanks,
Ionut