Re: [PATCH 3/4] KVM: x86/mmu: Add automatic prefaulting/prefetching for TDP MMU faults

[Marangoni, Marco]

On Thu, Oct 29, 2026, Sean Christopherson wrote:
>> From my perspective, using fast GUP has the problem that you cannot prefault
>> "around" your page, but just ahead of it, so I don't mind going with normal
>> GUP if it makes things easier for you.
>
> I think I'm missing something, why would fast GUP prevent prefaulting "behind"?
> E.g. is that effectively a limitation of the underlying backing store you're
> using, a fundamental limitation somewhere in mm/, or a limitation in KVM itself?

I didn't explain myself properly, sorry.

What I had in mind is that kvm_mmu_auto_prefault() returns on the first
kvm_tdp_page_prefault() failure, and starts from the ALIGN_DOWN'd window start
rather than from the faulting page.  So if a hole is below the faulting page,
the resident pages above it don't get prefetched either.

E.g. faulting at page 5, window [0..7].  If [4..7] are resident and [0..3] are
UFFD-missing, the loop fails on page 0 and nothing gets prefetched, even though
6 and 7 are right there.

It doesn't matter when nr_pages matches the UFFD chunk size and alignment, but
it would with smaller chunks or a non-linear access pattern.

FOLL_NOFAULT is fine by me; it just makes "hole" mean "not mapped" rather than
"not resident", so slightly more cases hit this.


> Hmm, right.  I deliberately hooked kvm_mmu_page_fault() instead of
> kvm_mmu_do_page_fault() so that prefetching wouldn't trigger auto-prefetch, but
> the existing shadow MMU prefetching does kick in for async #PFs, and it does make
> sense to prefetch in that case if we buy my argument (from this changelog) that
> async #PF should punch through dirty logging.
>
> Auto-prefetching for KVM_PRE_FAULT_MEMORY feels "wrong", but we do prefetch in
> that case for the shadow MMU, though I'm like 99.9% certain no one is running
> TDP with the shadow MMU these days, so I don't think we should consider that a
valid datapoint.
>
> Untested, but I think it'd be this: [...]

I measured the performances again with the change, and it fixes the issue with
the 8KiB prefetch window (on nested now ~14% faster than baseline). Nice.

________________________________________
From: Sean Christopherson <seanjc@xxxxxxxxxx>
Sent: Thursday, October 29, 2026 9:09 PM
To: Marangoni, Marco
Cc: kvm@xxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx; pbonzini@xxxxxxxxxx; cc-open-source@xxxxxxxxxx; James Houghton
Subject: RE: [EXTERNAL] [PATCH 3/4] KVM: x86/mmu: Add automatic prefaulting/prefetching for TDP MMU faults

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



On Thu, Oct 29, 2026, Marco Marangoni wrote:
> Hi Sean, thanks for the patch!
> I run some benchmarks on it, and it definitely helps! This is the time to fault 128 MiB linearly after a snapshot restore, with an UFFD page handler that faults chunks on demand; medians of 60 restores. Baseline is kvm-x86/next. Each entry is nr_pages: change vs baseline on the patched kernel; negative means faster.
>
> - UFFD 4 KiB handler (baseline 2228 ms) 0: +1%, 1: −5%, 2: −3%, 4: +1%, 8: −3%, 16: +1%, 32: 0%, 64: 0%, 128: −2%, 256: +1%, 512: +1%
> - UFFD 512 KiB handler (baseline 543 ms) 0: 0%, 1: 0%, 2: −36%, 4: −51%, 8: −59%, 16: −64%, 32: −71%, 64: −67%, 128: −68%, 256: −67%, 512: −66%
> - UFFD 2 MiB handler (baseline 513 ms) 0: 0%, 1: +1%, 2: −34%, 4: −50%, 8: −59%, 16: −64%, 32: −68%, 64: −68%, 128: −68%, 256: −68%, 512: −68%
>
> 8 is the series' default; 0 and 1 both disable prefetch and reproduce
> baseline. The 4 KiB row is within ±5%, i.e. noise.  It's measured on nested
> virtualization, so the improvement is probably less dramatic on metal. For
> context, my initial patch was around ~75% faster, so I'd say that
> performance-wise, your implementation is solid!
>
> It seems like a prefetch of 32 is the sweet spot, but I agree that with James
> that with the normal GUP it's probably better to keep it off or very low by
> default.

Yeah, agreed.  Triggering zswap or any other heavy operations could regress a
lot of configurations/workloads.

Hrm, but I don't want to lose support for guest_memfd without an hva.  Oh.  That's
a pre-existing bug, kvm_prefetch_pages() should reject kvm_memslot_is_gmem_only()
slots.  But that's a side topic.

Argh, FOLL_FAST_ONLY is internal-only.  But do we actually need it to be super
strictly fast-only?  Wouldn't FOLL_NOFAULT suffice?  IRQs are enabled so taking
mmap_sem is fine; it's not be ideal, but KVM will still try fast-GUP first, and
in practice performance is going to suffer no matter what if mmap_sem is
contended, e.g. arm64 *unconditionally* takes mmap_sem for read when handling
faults, and while it shows up in micro-benchmarks, e.g. mmu_stress_test, AFAIK
no one has complained otherwise.

So isn't the fix just this?

diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index b673c8130edd..1fcf4fedc0bb 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -4684,6 +4684,9 @@ static int __kvm_mmu_faultin_pfn(struct kvm_vcpu *vcpu,
 {
        unsigned int foll = fault->write ? FOLL_WRITE : 0;

+       if (fault->auto_prefetch)
+               foll |= FOLL_NOFAULT;
+
        if (fault->is_private || kvm_memslot_is_gmem_only(fault->slot))
                return kvm_mmu_faultin_pfn_gmem(vcpu, fault);

> From my perspective, using fast GUP has the problem that you cannot prefault
> "around" your page, but just ahead of it, so I don't mind going with normal
> GUP if it makes things easier for you.

I think I'm missing something, why would fast GUP prevent prefaulting "behind"?
E.g. is that effectively a limitation of the underlying backing store you're
using, a fundamental limitation somewhere in mm/, or a limitation in KVM itself?

> I'd say that most setup would know the size at which UFFD copies the memory,
> so having to explicitly configure is okay.
>
> For completeness, I also measured how much the patch improves at various UFFD chunks, with nr_pages at the same size:
>   - 4 KiB: 2285 ms → 2321 ms (np1), +2% (np1 is "off", noise)
>   - 8 KiB: 1435 ms → 1441 ms (np2), 0%
>   - 16 KiB: 1036 ms → 860 ms (np4), −17%
>   - 32 KiB: 743 ms → 469 ms (np8), −37%
>   - 64 KiB: 645 ms → 319 ms (np16), −50%
>   - 128 KiB: 587 ms → 242 ms (np32), −59%
>   - 256 KiB: 561 ms → 200 ms (np64), −64%
>   - 512 KiB: 543 ms → 165 ms (np128), −70%
>   - 1 MiB: 534 ms → 171 ms (np256), −68%
>   - 2 MiB: 523 ms → 151 ms (np512), −71%
>
> These measurement highlight a possible limitation (why didn't 8KiB improve?):
> it seems like the prefetching doesn't kick in the first time a page is
> getting faulted in the chunk, is that expected? I asked an LLM to
> investigate, and it seems to be because with async PF, the first fault
> doesn't reach the kvm_mmu_auto_prefault code path.

Hmm, right.  I deliberately hooked kvm_mmu_page_fault() instead of
kvm_mmu_do_page_fault() so that prefetching wouldn't trigger auto-prefetch, but
the existing shadow MMU prefetching does kick in for async #PFs, and it does make
sense to prefetch in that case if we buy my argument (from this changelog) that
async #PF should punch through dirty logging.

Auto-prefetching for KVM_PRE_FAULT_MEMORY feels "wrong", but we do prefetch in
that case for the shadow MMU, though I'm like 99.9% certain no one is running
TDP with the shadow MMU these days, so I don't think we should consider that a
valid datapoint.

Untested, but I think it'd be this:

diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index b673c8130edd..c0ff50c52ad5 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -5024,6 +5024,9 @@ enum kvm_prefetch_type {
        KVM_PREFETCH_AUTO,
 };

+static void kvm_mmu_auto_prefault(struct kvm_vcpu *vcpu, gpa_t start,
+                                 u64 error_code, u8 level);
+
 static int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
                                 u64 err, enum kvm_prefetch_type prefetch,
                                 int *emulation_type, u8 *level)
@@ -5098,6 +5101,9 @@ static int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
        if (level)
                *level = fault.goal_level;

+       if (r == RET_PF_FIXED && prefetch != KVM_PREFETCH_AUTO)
+               kvm_mmu_auto_prefault(vcpu, cr2_or_gpa, err, fault.goal_level);
+
        return r;
 }

or to disable it for KVM_PRE_FAULT_MEMORY:

diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index b673c8130edd..ee31a35f6a55 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -5021,9 +5021,13 @@ static int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *faul
 enum kvm_prefetch_type {
        KVM_PREFETCH_NONE,
        KVM_PREFETCH_MANUAL,
+       KVM_PREFETCH_ASYNC,
        KVM_PREFETCH_AUTO,
 };

+static void kvm_mmu_auto_prefault(struct kvm_vcpu *vcpu, gpa_t start,
+                                 u64 error_code, u8 level);
+
 static int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
                                 u64 err, enum kvm_prefetch_type prefetch,
                                 int *emulation_type, u8 *level)
@@ -5098,6 +5102,10 @@ static int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
        if (level)
                *level = fault.goal_level;

+       if (r == RET_PF_FIXED && prefetch != KVM_PREFETCH_AUTO &&
+           prefetch != KVM_PREFETCH_MANUAL)
+               kvm_mmu_auto_prefault(vcpu, cr2_or_gpa, err, fault.goal_level);
+
        return r;
 }

@@ -5215,7 +5223,7 @@ void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf *work)
                return;

        r = kvm_mmu_do_page_fault(vcpu, work->cr2_or_gpa, work->arch.error_code,
-                                 KVM_PREFETCH_MANUAL, NULL, NULL);
+                                 KVM_PREFETCH_ASYNC, NULL, NULL);

        /*
         * Account fixed page faults, otherwise they'll never be counted, but