Re: [PATCH RFC v2] KVM: arm64: nVHE: Flush all TLBs when freeing a VM's stage-2

[Marc Zyngier]

On Thu, 29 Oct 2026 15:34:48 +0000,
Francesco Boccola via B4 Relay <devnull+francesco.boccola.unina.it@xxxxxxxxxx> wrote:
> 
> From: Francesco Boccola <francesco.boccola@xxxxxxxx>
> 
> kvm_free_stage2_pgd() frees a VM's stage-2 page tables without invalidating
> the TLBs. That is correct: the VM's VMID is not reused before the next VMID
> rollover, and flush_context() then invalidates the entries of all VMIDs on
> all CPUs (TLBI ALLE1IS).
> 
> In nVHE mode, however, a CPU that ran a guest makes the host's broadcast
> TLB invalidations more expensive for the whole system until then, which
> with 16-bit VMIDs usually means until the host reboots.

Does it? Why? What permanent state is kept in the CPU TLBs that makes
invalidation slower?

> With stress-ng
> --tlb-shootdown on three CPUs of a cluster, the cyclictest maximum on a
> fourth, isolated CPU goes, before and after one guest:
> - Cortex-A53 (Zynq UltraScale+ KV260, no VHE, 6.1.70-rt): 53-72 us ->
>   83-164 us;
> - Cortex-A55 (RK3588, kvm-arm.mode=nvhe, kvmarm/next): 18-23 -> 78-97 us;
> - Cortex-A76 (RK3588, kvm-arm.mode=nvhe, kvmarm/next): 16-30 -> 74-80 us.
> It crosses clusters: after a guest on an A55, an isolated A76 goes from
> 14-18 us to 129-154 us. The same happens on kernel 6.12.113 (20-24 -> 44-49
> us on the A55) and on 7.1.9 with and without PREEMPT_RT.
> 
> One guest that turns its stage-1 MMU on, even for a few microseconds, is
> enough; a guest that keeps its MMU off is not. With VHE nothing changes.
> 
> The state is in the CPU that ran the guest: a local TLBI ALLE1 on that CPU
> at teardown removes the effect on every CPU, the same on all the other CPUs

I note that the hack you are adding in this patch is a broadcast TLBI,
not a local one.

> does not. Invalidating the dead VM's VMID (TLBI VMALLS12E1IS, as pKVM does
> at teardown) does not remove it either, whether stage 2 is disabled while
> it is issued (as __kvm_tlb_flush_vmid() does from the host's context) or
> enabled; nor does invalidating VMID 0 or the host's stage 1 (A53).

VMID 0 is not special. This is just a convention, and you could use
VMID $(RANDOM) as long as you use it consistently everywhere.

> 
> So do the rollover's invalidation when the stage 2 of a VM that has run is
> freed, in nVHE mode. That costs one TLB refill on every CPU per VM
> teardown; 500 VMs created, run and torn down one after the other take 32.1
> ms each with it and 33.4 ms without, and an isolated CPU meanwhile sees no
> change in latency (15 us against 13 us).

Not a lot in this report makes sense, I'm afraid. I'm not pretending
that what you are seeing isn't real (I haven't tried it), but without
a actual description of what the root cause of the problem is, we're
not going to randomly sprinkle TLB invalidations that are not
architecturally required.

	M.

-- 
Without deviation from the norm, progress is not possible.