Re: [REGRESSION] /proc/stat idle time exceeds wall clock since v7.2
From: Ahmed Shaltout
Date: Tue Sep 22 2026 - 12:02:42 EST
Hi Frederic,
Le Mon, Sep 21, 2026, Frederic Weisbecker a écrit :
> I can't manage to reproduce that, neither on latest mainline nor on 7.2.7
>
> Can you share your whole .config file?
Attached: config-7.2.5-1-default, taken from /boot/config-$(uname -r) on an
affected node. It is the stock openSUSE Tumbleweed/MicroOS package
kernel-default-7.2.5-1.1.x86_64, unmodified, no custom build. It differs from
config/x86_64/default in openSUSE:Factory/kernel-source at the 7.2.5 source
revision only in toolchain-detection and build-metadata lines (CC/AS/LD
version, LOCALVERSION, MODULE_SIG_KEY, CC_HAS_*), no functional option:
The unaffected control box runs kernel-default-7.1.8-1.1.x86_64 from the same
distribution, also stock, same VM type, same command line. Between the two
configs the only change among the scheduler, tick, accounting, preemption and
RCU options is CONFIG_SCHED_CACHE=y appearing in 7.2.5.
Environment, in case it is what your test box does not share:
- Every node is a KVM guest (Hetzner Cloud, QEMU, "AMD EPYC-Rome Processor",
8 vCPUs, one with 4), clocksource kvm-clock.
- No steal-time accounting at all: there is no kvm-stealtime line in the
boot log and the steal column has read exactly 0 on every node for the
75 hours since boot, so the hypervisor does not seem to expose the MSR.
- CONFIG_VIRT_CPU_ACCOUNTING_GEN=y, CONFIG_CONTEXT_TRACKING_IDLE=y and
CONFIG_NO_HZ_FULL=y are built in (SUSE default) with nohz_full= empty,
so the CPUs run idle dynticks, not full.
- CONFIG_HZ=1000, CONFIG_PREEMPT_DYNAMIC=y, psi=1 on the command line:
BOOT_IMAGE=/boot/vmlinuz-7.2.5-1-default root=UUID=... rd.timeout=60
rd.retry=45 quiet systemd.show_status=yes console=ttyS0,115200
console=tty0 ignition.platform.id=openstack security=selinux selinux=1
psi=1
It is fleet-wide, not one box. /proc/uptime idle as a percentage of
uptime x nr_cpus, read today on all eight nodes:
node kernel cpus uptime idle
cp-fsn1 7.2.5-1-default 8 75.0h 102.64%
cp-nbg1-pps 7.2.5-1-default 8 74.5h 102.03%
cp-nbg1-waa 7.2.5-1-default 8 74.2h 103.30%
cp-hel1 7.2.5-1-default 8 75.2h 97.59%
cp-hel1-b 7.2.5-1-default 8 75.2h 101.20%
worker-fsn1 7.2.5-1-default 8 74.0h 100.27%
monitoring 7.2.5-1-default 4 73.8h 102.75%
staging 7.1.8-1-default 8 852.4h 89.15%
cp-hel1 is the busiest node and stays under 100 only because real work
displaces it; the excess is an offset on idle, not a constant.
A 60 s delta over the per-CPU cpuN lines of /proc/stat on cp-nbg1-waa
(8 cores, 480.06 CPU-second ceiling):
user 10.84 nice 0.00 system 6.60 iowait 2.33 irq 0.00 softirq 2.65 steal 0.00
busy 22.42 = 4.67% of ceiling
idle 493.64 = 102.83% of ceiling
sum 516.06 = 107.50% of ceiling (excess 36.00, all of it in idle)
/proc/uptime's idle field moves with /proc/stat exactly (stat_idle /
uptime_idle = 1.0000 on both kernels), so the two files agree with each
other and only wall clock is exceeded. On 7.1.8 the modes sum to
0.992-0.997 of the ceiling over the same kind of window.
What did you reproduce on: bare metal or a KVM guest, and with
TICK_CPU_ACCOUNTING or VIRT_CPU_ACCOUNTING_GEN? I only have the version
bisection (7.1.8 good, 7.2.5 bad, same config family), not a commit. If it
helps I can boot kernel-vanilla and a 7.2.7 build on a throwaway VM of the
same type, or run any diagnostic you want on an affected node.
Thanks,
Ahmed
On Tue, Sep 22, 2026, 4:53 PM Frederic Weisbecker <frederic@xxxxxxxxxx> wrote:
Hi Ahmed,
Le Sun, Sep 20, 2026 at 03:14:22AM +0400, Ahmed Shaltout a écrit :
> Hello,
>
> Since moving from 7.1.8 to 7.2.5 (openSUSE MicroOS kernel-default), the idle
> time reported in /proc/stat exceeds the wall-clock time the CPUs have been up
> for. Idle alone accrues ~103% of the ceiling, and user+nice+system+idle+iowait
> +irq+softirq+steal sums to ~108% of it.
>
> This breaks the common `100 - rate(idle)` derivation of CPU utilisation, which
> goes negative on an idle-but-working machine. node_exporter, and anything built
> on it, is affected.
>
> #regzbot introduced: v7.1..v7.2
>
> Reproducer
> ==========
>
> No tooling needed; /proc/uptime shows it directly, since its second field is
> idle time and cannot legitimately exceed uptime x nr_cpus:
>
> read -r up idle < /proc/uptime && awk -v u="$up" -v i="$idle" -v c="$(nproc)" \
> 'BEGIN{printf "idle %.2f%% of ceiling\n", 100*i/(u*c)}'
>
> 7.2.5: idle 103.10% of ceiling
> 7.1.8: idle 89.13% of ceiling
I can't manage to reproduce that, neither on latest mainline nor on 7.2.7
Can you share your whole .config file?
Thanks.
--
Frederic Weisbecker
SUSE Labs
Attachment:
config-7.2.5-1-default
Description: Binary data