[RFC PATCH 25/55] WIP: KVM: per-plane memory overlays, richer protection-fault exits
From: David Woodhouse
Date: Sat Oct 17 2026 - 08:40:12 EST
From: David Woodhouse <dwmw@xxxxxxxxxxxx>
The pieces the VSM contract tests need:
- KVM_SET_PLANE_MEMORY_OVERLAY maps one page of userspace memory over
a GPA for a single plane. TLFS overlay pages (SynIC SIMP in
particular) show each VTL a private page while the other VTLs keep
seeing the guest RAM underneath; per-plane EPT roots make a
per-plane pfn override in the fault path sufficient. Mappings for a
plane with overlays stay 4KiB so a neighbouring fault's huge page
cannot span an overlaid gfn, and kvm_tdp_mmu_recover_huge_pages()
skips nonzero-plane roots so a later promotion cannot widen the
plane's attribute or overlay clamps back out (planes keep no lpage
tracking by design; their fault path rebuilds each mapping at the
largest level the plane allows). The overlay update runs under the
kvm_mmu_invalidate_begin()/end() protocol with the gfn registered,
as the mmu notifiers do, so a concurrent lockless fault cannot
install an SPTE for the old backing after the zap.
- SynIC message delivery reads and writes the overlay through the same
lookup, and fails with -EFAULT when a higher plane's SIMP page has
no overlay installed yet, rather than falling through to
kvm_vcpu_write_guest_page() and corrupting underlay RAM that plane 0
owns; the sender retries once the overlay appears. A non-zero
plane's SynIC exits carry the plane instead of being suppressed, so
the VMM can keep its plane-0 device state separate.
- A plane-protection fault reflected as KVM_EXIT_MEMORY_FAULT now
carries the guest-linear address and the faulting instruction's
length and bytes (via VM_EXIT_INSTRUCTION_LEN and a guest-virt
read), which the TLFS GPA-intercept message requires. LLDT joins
LGDT/LIDT in the descriptor-write divert, reporting the written
selector.
Signed-off-by: David Woodhouse <dwmw@xxxxxxxxxxxx>
Assisted-by: LLM
---
Documentation/virt/kvm/api.rst | 62 ++++++++
arch/x86/kvm/hyperv.c | 82 +++++++---
arch/x86/kvm/mmu/mmu.c | 29 +++-
arch/x86/kvm/mmu/tdp_mmu.c | 23 ++-
arch/x86/kvm/vmx/vmx.c | 51 ++++++-
arch/x86/kvm/x86.c | 14 ++
include/linux/kvm_host.h | 17 +++
include/uapi/linux/kvm.h | 17 +++
virt/kvm/kvm_main.c | 271 +++++++++++++++++++++++++++++++++
9 files changed, 541 insertions(+), 25 deletions(-)
diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index 3ef57b3b050b..215547802377 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -6657,6 +6657,38 @@ stop policy.
The run state is distinct from KVM_GET/SET_MP_STATE, which describes
the architectural state of a vCPU within its plane.
+4.149 KVM_SET_PLANE_MEMORY_OVERLAY
+----------------------------------
+
+:Capability: KVM_CAP_PLANE_MEMORY_OVERLAYS
+:Architectures: x86
+:Type: vm ioctl
+:Parameters: struct kvm_plane_memory_overlay (in)
+:Returns: 0 on success, <0 on error
+
+::
+
+ struct kvm_plane_memory_overlay {
+ __u32 plane;
+ __u32 flags;
+ __u64 gfn;
+ __u64 uaddr; /* page-aligned; 0 removes the overlay */
+ };
+
+Installs, replaces or removes a memory overlay for a single guest page
+frame on one plane. While an overlay is installed, translations of
+'gfn' for vCPUs of 'plane' resolve to the (pinned) page at 'uaddr' in
+the calling process instead of to the memslot backing; other planes are
+unaffected. Hyper-V VSM uses this for per-VTL overlay pages such as
+the SynIC message and event-flags pages.
+
+'flags' must be zero and 'uaddr' must be page-aligned. At most
+KVM_CHECK_EXTENSION(KVM_CAP_PLANE_MEMORY_OVERLAYS) distinct gfns can
+carry an overlay per plane at any one time; further installs fail with
+-ENOSPC until an overlay is removed. An overlay cannot be installed
+over a private or guest_memfd-only gfn (-EINVAL). Passing 'uaddr' == 0
+removes the overlay for 'gfn', if any.
+
5. The kvm_run structure
========================
@@ -7387,9 +7419,15 @@ spec refer, https://github.com/riscv/riscv-sbi-doc.
#define KVM_MEMORY_EXIT_FLAG_WRITE (1ULL << 1)
#define KVM_MEMORY_EXIT_FLAG_EXEC (1ULL << 2)
#define KVM_MEMORY_EXIT_FLAG_PRIVATE (1ULL << 3)
+ #define KVM_MEMORY_EXIT_FLAG_GVA_VALID (1ULL << 4)
__u64 flags;
__u64 gpa;
__u64 size;
+ /* set on plane-protection faults when available */
+ __u64 gva;
+ __u32 insn_len;
+ __u8 insn_bytes[16];
+ __u32 pad;
} memory_fault;
KVM_EXIT_MEMORY_FAULT indicates the vCPU has encountered a memory fault that
@@ -7402,6 +7440,14 @@ describes properties of the faulting access that are likely pertinent:
- KVM_MEMORY_EXIT_FLAG_PRIVATE - When set, indicates the memory fault occurred
on a private memory access. When clear, indicates the fault occurred on a
shared access.
+ - KVM_MEMORY_EXIT_FLAG_GVA_VALID - When set, 'gva' holds the faulting guest
+ virtual address.
+
+The 'gva', 'insn_len' and 'insn_bytes' fields exist only on kernels
+which advertise KVM_CAP_PLANE_MEMORY_OVERLAYS (see section 8.48). On
+plane memory-protection faults, 'insn_len' and 'insn_bytes' carry a
+best-effort copy of the faulting instruction when KVM could fetch it;
+an 'insn_len' of 0 means no instruction bytes are available.
Note! KVM_EXIT_MEMORY_FAULT is unique among all KVM exit reasons in that it
accompanies a return code of '-1', not '0'! errno will always be set to EFAULT
@@ -9676,6 +9722,22 @@ take care to differentiate between these cases.
The presence of this capability indicates that the nested KVM guest can
start in ESA mode.
+8.48 KVM_CAP_PLANE_MEMORY_OVERLAYS
+----------------------------------
+
+:Architectures: x86
+
+A positive return from KVM_CHECK_EXTENSION(KVM_CAP_PLANE_MEMORY_OVERLAYS)
+announces two things:
+
+ - The KVM_SET_PLANE_MEMORY_OVERLAY vm ioctl (section 4.149). The
+ returned value is the number of gfns which can carry an overlay
+ simultaneously on each plane.
+
+ - The extended KVM_EXIT_MEMORY_FAULT layout: 'memory_fault' carries
+ the additional 'gva', 'insn_len' and 'insn_bytes' fields, with
+ KVM_MEMORY_EXIT_FLAG_GVA_VALID qualifying 'gva' (see section 5).
+
9. Known KVM API problems
=========================
diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c
index 4551e011bf56..d4e44ae3f6e8 100644
--- a/arch/x86/kvm/hyperv.c
+++ b/arch/x86/kvm/hyperv.c
@@ -247,16 +247,8 @@ static void synic_exit(struct kvm_vcpu_hv_synic *synic, u32 msr)
struct kvm_vcpu *vcpu = hv_synic_to_vcpu(synic);
struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);
- /*
- * Userspace's SynIC device models plane 0 only; a higher plane's
- * exit would be mistaken for a plane-0 page move and relocate the
- * overlay the VMM keeps for SIMP/SIEFP. Higher-plane SynICs are
- * handled fully in-kernel.
- */
- if (vcpu->plane_level)
- return;
-
hv_vcpu->exit.type = KVM_EXIT_HYPERV_SYNIC;
+ hv_vcpu->exit.plane = vcpu->plane_level;
hv_vcpu->exit.u.synic.msr = msr;
hv_vcpu->exit.u.synic.control = synic->control;
hv_vcpu->exit.u.synic.evt_page = synic->evt_page;
@@ -358,6 +350,7 @@ static void syndbg_exit(struct kvm_vcpu *vcpu, u32 msr)
struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);
hv_vcpu->exit.type = KVM_EXIT_HYPERV_SYNDBG;
+ hv_vcpu->exit.plane = vcpu->plane_level;
hv_vcpu->exit.u.syndbg.msr = msr;
hv_vcpu->exit.u.syndbg.control = syndbg->control.control;
hv_vcpu->exit.u.syndbg.send_page = syndbg->control.send_page;
@@ -783,6 +776,51 @@ static int stimer_get_count(struct kvm_vcpu_hv_stimer *stimer, u64 *pcount)
return 0;
}
+/*
+ * SIMP accessors: a plane overlay page, when installed by userspace,
+ * carries the live message page for this plane; the memslot holds only
+ * the underlay the other planes see.
+ */
+static int synic_msg_page_read(struct kvm_vcpu *vcpu, gfn_t gfn, void *data,
+ int offset, int len)
+{
+ unsigned long uaddr = kvm_plane_overlay_uaddr(vcpu->plane, gfn);
+
+ if (uaddr)
+ return copy_from_user(data, (void __user *)(uaddr + offset),
+ len) ? -EFAULT : 0;
+ /*
+ * Mirror the write side: without an overlay a higher plane's
+ * message page is backed by underlay memory that plane 0 controls.
+ * Deciding deliverability from plane-0-writable bytes would let a
+ * lower VTL plant a non-HVMSG_NONE type and silently swallow a
+ * no_retry (periodic stimer) message; fail so the delivery is
+ * retried once the overlay appears.
+ */
+ if (vcpu->plane_level)
+ return -EFAULT;
+ return kvm_vcpu_read_guest_page(vcpu, gfn, data, offset, len);
+}
+
+static int synic_msg_page_write(struct kvm_vcpu *vcpu, gfn_t gfn,
+ const void *data, int offset, int len)
+{
+ unsigned long uaddr = kvm_plane_overlay_uaddr(vcpu->plane, gfn);
+
+ if (uaddr)
+ return copy_to_user((void __user *)(uaddr + offset), data,
+ len) ? -EFAULT : 0;
+ /*
+ * Never write a higher plane's messages into guest RAM: without an
+ * overlay installed that memory is the underlay every other plane
+ * sees, and silently corrupting plane-0-owned pages is far worse
+ * than retrying the message once the overlay appears.
+ */
+ if (vcpu->plane_level)
+ return -EFAULT;
+ return kvm_vcpu_write_guest_page(vcpu, gfn, data, offset, len);
+}
+
static int synic_deliver_msg(struct kvm_vcpu_hv_synic *synic, u32 sint,
struct hv_message *src_msg, bool no_retry)
{
@@ -803,10 +841,10 @@ static int synic_deliver_msg(struct kvm_vcpu_hv_synic *synic, u32 sint,
* is only called in vcpu context so the entire update is atomic from
* guest POV and thus the exact order here doesn't matter.
*/
- r = kvm_vcpu_read_guest_page(vcpu, msg_page_gfn, &hv_hdr.message_type,
- msg_off + offsetof(struct hv_message,
- header.message_type),
- sizeof(hv_hdr.message_type));
+ r = synic_msg_page_read(vcpu, msg_page_gfn, &hv_hdr.message_type,
+ msg_off + offsetof(struct hv_message,
+ header.message_type),
+ sizeof(hv_hdr.message_type));
if (r < 0)
return r;
@@ -815,20 +853,20 @@ static int synic_deliver_msg(struct kvm_vcpu_hv_synic *synic, u32 sint,
return 0;
hv_hdr.message_flags.msg_pending = 1;
- r = kvm_vcpu_write_guest_page(vcpu, msg_page_gfn,
- &hv_hdr.message_flags,
- msg_off +
- offsetof(struct hv_message,
- header.message_flags),
- sizeof(hv_hdr.message_flags));
+ r = synic_msg_page_write(vcpu, msg_page_gfn,
+ &hv_hdr.message_flags,
+ msg_off +
+ offsetof(struct hv_message,
+ header.message_flags),
+ sizeof(hv_hdr.message_flags));
if (r < 0)
return r;
return -EAGAIN;
}
- r = kvm_vcpu_write_guest_page(vcpu, msg_page_gfn, src_msg, msg_off,
- sizeof(src_msg->header) +
- src_msg->header.payload_size);
+ r = synic_msg_page_write(vcpu, msg_page_gfn, src_msg, msg_off,
+ sizeof(src_msg->header) +
+ src_msg->header.payload_size);
if (r < 0)
return r;
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index cdebb5c324fe..4b11631d9e99 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -4620,7 +4620,13 @@ static int kvm_mmu_faultin_pfn_gmem(struct kvm_vcpu *vcpu,
if (fault->slot->flags & KVM_MEM_READONLY)
fault->host_access &= ~ACC_WRITE_MASK;
- fault->max_level = kvm_max_level_for_order(max_order);
+ /*
+ * Only ever narrow max_level: the overlay check in
+ * __kvm_mmu_faultin_pfn() may already have clamped it to 4KiB, and
+ * a huge gmem mapping must not span an overlaid gfn.
+ */
+ fault->max_level = min(fault->max_level,
+ kvm_max_level_for_order(max_order));
return RET_PF_CONTINUE;
}
@@ -4682,8 +4688,29 @@ static int __kvm_mmu_faultin_pfn(struct kvm_vcpu *vcpu,
struct kvm_page_fault *fault)
{
unsigned int foll = fault->write ? FOLL_WRITE : 0;
+ struct page *overlay;
bool writable;
+ /*
+ * A plane overlay page (VSM SynIC SIMP/SIEFP etc.) replaces the
+ * memslot translation for this plane only. The page is pinned
+ * for the overlay's lifetime, so no refcounting here.
+ */
+ /*
+ * Keep this plane's mappings at 4KiB while it has overlays: a huge
+ * mapping established by a neighbouring gfn's fault would span the
+ * overlaid gfn and bypass the override below.
+ */
+ if (READ_ONCE(vcpu->plane->nr_overlays)) {
+ fault->max_level = PG_LEVEL_4K;
+
+ overlay = kvm_plane_overlay_page(vcpu->plane, fault->gfn);
+ if (overlay) {
+ fault->pfn = page_to_pfn(overlay);
+ return RET_PF_CONTINUE;
+ }
+ }
+
if (fault->is_private || kvm_memslot_is_gmem_only(fault->slot))
return kvm_mmu_faultin_pfn_gmem(vcpu, fault);
diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c
index 1e56a4784160..d5d0b2bc547c 100644
--- a/arch/x86/kvm/mmu/tdp_mmu.c
+++ b/arch/x86/kvm/mmu/tdp_mmu.c
@@ -1871,8 +1871,29 @@ void kvm_tdp_mmu_recover_huge_pages(struct kvm *kvm,
struct kvm_mmu_page *root;
lockdep_assert_held_read(&kvm->mmu_lock);
- for_each_valid_tdp_mmu_root_yield_safe(kvm, root, slot->as_id)
+ for_each_valid_tdp_mmu_root_yield_safe(kvm, root, slot->as_id) {
+ /*
+ * kvm_mmu_max_mapping_level() consults only plane-0 state
+ * (lpage_info and plane-0 private attributes). Nonzero
+ * planes carry per-plane memory attributes and memory
+ * overlays whose fault-time 4KiB clamps a huge SPTE would
+ * undo, extending one page's protections or overlay
+ * backing over gfns the plane maps differently. Leave
+ * those roots alone; their fault path rebuilds mappings at
+ * the largest level the plane allows.
+ *
+ * Plane 0 can carry overlays too (kvm->planes[0] always
+ * exists and KVM_SET_PLANE_MEMORY_OVERLAY accepts plane 0),
+ * and the same reasoning applies: a recovered huge SPTE
+ * would either span an overlaid gfn with the memslot
+ * backing, or huge-align the pinned overlay page's pfn and
+ * map ~2MiB of unrelated host memory around it.
+ */
+ if (root->role.plane ||
+ READ_ONCE(kvm->planes[root->role.plane]->nr_overlays))
+ continue;
recover_huge_pages_range(kvm, root, slot);
+ }
}
/*
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index e1c5f1608b09..248d799b46e9 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -70,6 +70,7 @@
#include "vmcs12.h"
#include "vmx.h"
#include "x86.h"
+#include "kvm_emulate.h"
#include "x86_ops.h"
#include "smm.h"
#include "vmx_onhyperv.h"
@@ -5949,6 +5950,7 @@ static int handle_ept_violation(struct kvm_vcpu *vcpu)
{
unsigned long exit_qualification = vmx_get_exit_qual(vcpu);
gpa_t gpa;
+ int ret;
/*
* EPT violation happened while executing iret from NMI,
@@ -5975,7 +5977,54 @@ static int handle_ept_violation(struct kvm_vcpu *vcpu)
if (unlikely(allow_smaller_maxphyaddr && !kvm_vcpu_is_legal_gpa(vcpu, gpa)))
return kvm_emulate_instruction(vcpu, 0);
- return __vmx_handle_ept_violation(vcpu, gpa, exit_qualification);
+ ret = __vmx_handle_ept_violation(vcpu, gpa, exit_qualification);
+
+ /*
+ * A plane-protection fault reflected to userspace becomes a TLFS
+ * GPA intercept message, which carries the guest-linear address
+ * and the faulting instruction's bytes. Harvest them while the
+ * exit state is still at hand.
+ */
+ if (ret == -EFAULT &&
+ vcpu->run->exit_reason == KVM_EXIT_MEMORY_FAULT &&
+ (vcpu->run->memory_fault.flags & (KVM_MEMORY_EXIT_FLAG_READ |
+ KVM_MEMORY_EXIT_FLAG_WRITE |
+ KVM_MEMORY_EXIT_FLAG_EXEC))) {
+ unsigned int len;
+
+ if (exit_qualification & EPT_VIOLATION_GVA_IS_VALID) {
+ vcpu->run->memory_fault.gva =
+ vmcs_readl(GUEST_LINEAR_ADDRESS);
+ vcpu->run->memory_fault.flags |=
+ KVM_MEMORY_EXIT_FLAG_GVA_VALID;
+ }
+ /*
+ * Architecturally undefined for EPT violations, but the
+ * hardware provides the faulting (for a fetch: the
+ * fetching) instruction's length here, and the TLFS
+ * message carries exactly that.
+ */
+ len = vmcs_read32(VM_EXIT_INSTRUCTION_LEN);
+ len = min_t(unsigned int, len,
+ sizeof(vcpu->run->memory_fault.insn_bytes));
+ vcpu->run->memory_fault.insn_len = len;
+ if (!(exit_qualification & EPT_VIOLATION_ACC_INSTR)) {
+ struct x86_exception e;
+
+ /*
+ * kvm_read_guest_virt() takes a linear address: RIP
+ * is CS-relative, and the two coincide only when
+ * CS.base is zero (64-bit mode, or a flat 32-bit
+ * code segment).
+ */
+ if (kvm_read_guest_virt(vcpu, kvm_get_linear_rip(vcpu),
+ vcpu->run->memory_fault.insn_bytes,
+ len, &e))
+ memset(vcpu->run->memory_fault.insn_bytes, 0,
+ sizeof(vcpu->run->memory_fault.insn_bytes));
+ }
+ }
+ return ret;
}
static int handle_ept_misconfig(struct kvm_vcpu *vcpu)
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 4b12ec7ee100..10e14ab3e2ca 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -8858,8 +8858,10 @@ static bool hv_divert_dt_write(struct x86_emulate_ctxt *ctxt, u32 hv_reg,
#define HV_CR_INTERCEPT_GDTR_WRITE BIT_ULL(15)
#define HV_CR_INTERCEPT_IDTR_WRITE BIT_ULL(16)
+#define HV_CR_INTERCEPT_LDTR_WRITE BIT_ULL(17)
#define HV_X64_REGISTER_IDTR_ID 0x00070000
#define HV_X64_REGISTER_GDTR_ID 0x00070001
+#define HV_X64_REGISTER_LDTR_ID 0x00060006
static void emulator_set_gdt(struct x86_emulate_ctxt *ctxt, struct desc_ptr *dt)
{
@@ -8947,6 +8949,18 @@ static void emulator_set_segment(struct x86_emulate_ctxt *ctxt, u16 selector,
var.unusable = !var.present;
var.padding = 0;
+ if (seg == VCPU_SREG_LDTR) {
+ struct desc_ptr sel_as_dt = {
+ .address = selector,
+ .size = 0,
+ };
+
+ /* The intercept message carries the written selector. */
+ if (hv_divert_dt_write(ctxt, HV_X64_REGISTER_LDTR_ID,
+ &sel_as_dt, HV_CR_INTERCEPT_LDTR_WRITE))
+ return;
+ }
+
kvm_set_segment(vcpu, &var, seg);
return;
}
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index a76615f2c3dc..cb8c4798df12 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -927,9 +927,26 @@ struct kvm_plane {
struct xarray mem_attrs;
#endif
+ /*
+ * VSM overlay pages: per-plane gfn -> pinned user page overriding
+ * the memslot translation for this plane only. Written under
+ * kvm->slots_lock; readers tolerate a transient stale view, the
+ * post-update zap makes the EPT coherent.
+ */
+#define KVM_PLANE_MAX_OVERLAYS 64
+ struct kvm_plane_overlay {
+ gfn_t gfn;
+ unsigned long uaddr;
+ struct page *page;
+ } overlays[KVM_PLANE_MAX_OVERLAYS];
+ int nr_overlays;
+
struct kvm_arch_plane arch;
};
+struct page *kvm_plane_overlay_page(struct kvm_plane *plane, gfn_t gfn);
+unsigned long kvm_plane_overlay_uaddr(struct kvm_plane *plane, gfn_t gfn);
+
struct kvm {
#ifdef KVM_HAVE_MMU_RWLOCK
rwlock_t mmu_lock;
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 9e453f215e68..843cef8f4cbf 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -480,9 +480,15 @@ struct kvm_run {
#define KVM_MEMORY_EXIT_FLAG_WRITE (1ULL << 1)
#define KVM_MEMORY_EXIT_FLAG_EXEC (1ULL << 2)
#define KVM_MEMORY_EXIT_FLAG_PRIVATE (1ULL << 3)
+#define KVM_MEMORY_EXIT_FLAG_GVA_VALID (1ULL << 4)
__u64 flags;
__u64 gpa;
__u64 size;
+ /* set on plane-protection faults when available */
+ __u64 gva;
+ __u32 insn_len;
+ __u8 insn_bytes[16];
+ __u32 pad;
} memory_fault;
/* KVM_EXIT_TDX */
struct {
@@ -1038,6 +1044,7 @@ struct kvm_enable_cap {
#define KVM_CAP_HCALL_FAULT_EXIT 252
#define KVM_CAP_HYPERV_VTL_SWITCH 253
#define KVM_CAP_HYPERV_REG_INTERCEPT 254
+#define KVM_CAP_PLANE_MEMORY_OVERLAYS 257
struct kvm_irq_routing_irqchip {
__u32 irqchip;
@@ -1394,6 +1401,16 @@ struct kvm_s390_keyop {
#define KVM_CREATE_PLANE _IO(KVMIO, 0xe4)
#define KVM_GET_PLANE_RUN_STATE _IOR(KVMIO, 0xe5, __u32)
#define KVM_SET_PLANE_RUN_STATE _IOW(KVMIO, 0xe6, __u32)
+
+/* Map one page of userspace memory over a GPA for a single plane (VSM
+ * overlay pages: SynIC SIMP/SIEFP, hypercall page, VP assist page). */
+struct kvm_plane_memory_overlay {
+ __u32 plane;
+ __u32 flags;
+ __u64 gfn;
+ __u64 uaddr; /* page-aligned; 0 removes the overlay */
+};
+#define KVM_SET_PLANE_MEMORY_OVERLAY _IOW(KVMIO, 0xe7, struct kvm_plane_memory_overlay)
#define KVM_PLANE_RUN_STATE_STOPPED 0
#define KVM_PLANE_RUN_STATE_RUNNABLE 1
#define KVM_PLANE_RUN_STATE_KERNEL_STOPPED 2
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 4197496f7176..5856672bbf6d 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -1251,6 +1251,14 @@ static struct kvm_plane *kvm_create_plane(struct kvm *kvm, unsigned plane_level)
static void kvm_destroy_one_plane(struct kvm_plane *plane)
{
+ int i;
+
+ for (i = 0; i < plane->nr_overlays; i++)
+ if (plane->overlays[i].page)
+ unpin_user_pages_dirty_lock(&plane->overlays[i].page,
+ 1, true);
+ plane->nr_overlays = 0;
+
#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
xa_destroy(&plane->mem_attrs);
#endif
@@ -5629,6 +5637,8 @@ static int kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg)
* VM actually has.
*/
return KVM_MAX_PLANES;
+ case KVM_CAP_PLANE_MEMORY_OVERLAYS:
+ return KVM_PLANE_MAX_OVERLAYS;
default:
break;
}
@@ -5896,6 +5906,258 @@ do { \
sizeof_field(struct kvm_userspace_memory_region2, field)); \
} while (0)
+struct page *kvm_plane_overlay_page(struct kvm_plane *plane, gfn_t gfn)
+{
+ int i, n = READ_ONCE(plane->nr_overlays);
+
+ /* pairs with the smp_wmb() publishing a new entry */
+ smp_rmb();
+ for (i = 0; i < n; i++) {
+ if (READ_ONCE(plane->overlays[i].gfn) != gfn)
+ continue;
+ /* pairs with the smp_store_release() updating an entry */
+ return smp_load_acquire(&plane->overlays[i].page);
+ }
+ return NULL;
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_plane_overlay_page);
+
+unsigned long kvm_plane_overlay_uaddr(struct kvm_plane *plane, gfn_t gfn)
+{
+ int i, n = READ_ONCE(plane->nr_overlays);
+
+ smp_rmb();
+ for (i = 0; i < n; i++) {
+ if (READ_ONCE(plane->overlays[i].gfn) != gfn)
+ continue;
+ /*
+ * The acquire pairs with the smp_store_release() updating
+ * an entry: a reader which observes the new page also
+ * observes the uaddr written before it.
+ */
+ if (smp_load_acquire(&plane->overlays[i].page))
+ return READ_ONCE(plane->overlays[i].uaddr);
+ return 0;
+ }
+ return 0;
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_plane_overlay_uaddr);
+
+/*
+ * The invalidation window registered around an overlay update must
+ * cover every gfn whose in-flight fault could install a huge leaf
+ * spanning the overlaid gfn, i.e. the naturally-aligned region of the
+ * widest stage-2 huge mapping any plane-capable architecture supports
+ * (a 1GiB page on x86). A narrower window lets a racing fault on a
+ * neighbouring gfn, which sampled nr_overlays and mmu_invalidate_seq
+ * before the update, pass mmu_invalidate_retry_gfn() and install a
+ * huge SPTE over the overlay after the zap.
+ */
+#define KVM_PLANE_OVERLAY_INVAL_PAGES (SZ_1G >> PAGE_SHIFT)
+
+static void kvm_plane_overlay_invalidate_add(struct kvm *kvm, gfn_t gfn)
+{
+ gfn_t start = ALIGN_DOWN(gfn, KVM_PLANE_OVERLAY_INVAL_PAGES);
+
+ kvm_mmu_invalidate_range_add(kvm, start,
+ start + KVM_PLANE_OVERLAY_INVAL_PAGES);
+}
+
+/*
+ * Drop every translation of @gfn, in all address spaces and all planes'
+ * roots. A huge leaf spanning the gfn intersects the single-page range
+ * and is zapped with it. Hardware TLBs are not flushed here.
+ */
+static void kvm_plane_overlay_unmap(struct kvm *kvm, gfn_t gfn)
+{
+#ifdef CONFIG_KVM_GENERIC_MMU_NOTIFIER
+ struct kvm_memslot_iter iter;
+ struct kvm_memslots *slots;
+ int i;
+
+ KVM_MMU_LOCK(kvm);
+ for (i = 0; i < kvm_arch_nr_memslot_as_ids(kvm); i++) {
+ slots = __kvm_memslots(kvm, i);
+
+ kvm_for_each_memslot_in_gfn_range(&iter, slots, gfn, gfn + 1) {
+ struct kvm_gfn_range range = {
+ .slot = iter.slot,
+ .start = gfn,
+ .end = gfn + 1,
+ .attr_filter = KVM_FILTER_PRIVATE |
+ KVM_FILTER_SHARED,
+ .may_block = true,
+ };
+
+ kvm_unmap_gfn_range(kvm, &range);
+ }
+ }
+ KVM_MMU_UNLOCK(kvm);
+#else
+ kvm_arch_flush_shadow_all(kvm);
+#endif
+}
+
+static int kvm_vm_ioctl_set_plane_overlay(struct kvm *kvm,
+ struct kvm_plane_memory_overlay *ov)
+{
+ struct kvm_plane_overlay *e = NULL;
+ struct page *page = NULL, *old_page = NULL;
+ struct kvm_plane *plane;
+ bool moved = false;
+ gfn_t moved_gfn = 0;
+ int i, r;
+
+ if (ov->flags || ov->plane >= KVM_MAX_PLANES)
+ return -EINVAL;
+ if (ov->uaddr & ~PAGE_MASK)
+ return -EINVAL;
+ plane = kvm->planes[ov->plane];
+ if (!plane)
+ return -EINVAL;
+
+ if (ov->uaddr) {
+ r = pin_user_pages_fast(ov->uaddr, 1,
+ FOLL_WRITE | FOLL_LONGTERM, &page);
+ if (r != 1)
+ return r < 0 ? r : -EFAULT;
+ }
+
+ mutex_lock(&kvm->slots_lock);
+ for (i = 0; i < plane->nr_overlays; i++) {
+ if (plane->overlays[i].gfn == ov->gfn) {
+ e = &plane->overlays[i];
+ break;
+ }
+ }
+
+ /*
+ * Removing an overlay which was never installed changes nothing;
+ * don't zap and flush for it.
+ */
+ if (!ov->uaddr && !e) {
+ mutex_unlock(&kvm->slots_lock);
+ return 0;
+ }
+
+ /*
+ * guest_memfd-backed and private gfns have their own backing
+ * contract which an ordinary pinned user page cannot substitute
+ * for, and kvm_mmu_faultin_pfn_gmem() does not consult overlays.
+ * Both the private attribute and the memslot flags are stable
+ * under slots_lock.
+ */
+ if (ov->uaddr) {
+ struct kvm_memory_slot *slot = gfn_to_memslot(kvm, ov->gfn);
+
+ if ((slot && kvm_memslot_is_gmem_only(slot)) ||
+ kvm_mem_is_private(kvm, ov->gfn)) {
+ mutex_unlock(&kvm->slots_lock);
+ unpin_user_page(page);
+ return -EINVAL;
+ }
+ }
+
+ if (ov->uaddr) {
+ if (!e) {
+ if (plane->nr_overlays >= KVM_PLANE_MAX_OVERLAYS) {
+ mutex_unlock(&kvm->slots_lock);
+ unpin_user_page(page);
+ return -ENOSPC;
+ }
+ e = &plane->overlays[plane->nr_overlays];
+ e->gfn = ov->gfn;
+ e->uaddr = ov->uaddr;
+ e->page = page;
+ /*
+ * Publish the fully-initialised entry before
+ * raising the count; pairs with the smp_rmb() in
+ * the lockless readers.
+ */
+ smp_wmb();
+ WRITE_ONCE(plane->nr_overlays, plane->nr_overlays + 1);
+ } else {
+ old_page = e->page;
+ WRITE_ONCE(e->uaddr, ov->uaddr);
+ /*
+ * Order the uaddr store before the page store; a
+ * reader which acquires the new page must see the
+ * uaddr it belongs to.
+ */
+ smp_store_release(&e->page, page);
+ }
+ } else if (e) {
+ struct kvm_plane_overlay *tail =
+ &plane->overlays[plane->nr_overlays - 1];
+
+ old_page = e->page;
+ WRITE_ONCE(e->page, NULL);
+ WRITE_ONCE(e->uaddr, 0);
+
+ /*
+ * Reclaim the slot: move the tail entry into the vacated
+ * one and shrink the array. A lockless reader scanning
+ * with the old count can miss the moved entry entirely
+ * (it may already have passed the vacated slot), so the
+ * moved gfn is added to the invalidation window below and
+ * any such in-flight fault retries. The cleared tail is
+ * harmless to a stale-count reader: the live copy sits at
+ * a lower index and is found first.
+ */
+ if (e != tail) {
+ moved = true;
+ moved_gfn = tail->gfn;
+ WRITE_ONCE(e->gfn, tail->gfn);
+ WRITE_ONCE(e->uaddr, tail->uaddr);
+ /* pairs with smp_load_acquire() in the readers */
+ smp_store_release(&e->page, tail->page);
+ }
+ WRITE_ONCE(plane->nr_overlays, plane->nr_overlays - 1);
+ WRITE_ONCE(tail->page, NULL);
+ WRITE_ONCE(tail->uaddr, 0);
+ WRITE_ONCE(tail->gfn, 0);
+ }
+ /*
+ * Bump the invalidation sequence across the update and zap: a
+ * concurrent fault resolves the overlay pfn locklessly before
+ * taking mmu_lock, and without a sequence change
+ * is_page_fault_stale() would let it install an SPTE for the OLD
+ * backing after the zap, mapping a page this function is about
+ * to unpin.
+ */
+ KVM_MMU_LOCK(kvm);
+ kvm_mmu_invalidate_begin(kvm);
+ kvm_plane_overlay_invalidate_add(kvm, ov->gfn);
+ if (moved)
+ kvm_plane_overlay_invalidate_add(kvm, moved_gfn);
+ KVM_MMU_UNLOCK(kvm);
+
+ mutex_unlock(&kvm->slots_lock);
+
+ /*
+ * Drop any translation mapping the old or new backing. The unmap
+ * removes the SPTEs but does not flush hardware TLBs: a vCPU that
+ * mapped the gfn before the update still holds a translation to
+ * the old backing in its TLB (tagged by the live ASID on SVM,
+ * which nothing else invalidates). Flush before the old page is
+ * unpinned: KVM_REQ_TLB_FLUSH is a per-plane request, so the
+ * fan-out in kvm_make_vcpu_request() reaches every plane twin and
+ * each flushes its own hardware context.
+ */
+ kvm_plane_overlay_unmap(kvm, ov->gfn);
+ if (moved)
+ kvm_plane_overlay_unmap(kvm, moved_gfn);
+ kvm_flush_remote_tlbs(kvm);
+
+ KVM_MMU_LOCK(kvm);
+ kvm_mmu_invalidate_end(kvm);
+ KVM_MMU_UNLOCK(kvm);
+
+ if (old_page)
+ unpin_user_pages_dirty_lock(&old_page, 1, true);
+ return 0;
+}
+
static long kvm_vm_ioctl(struct file *filp,
unsigned int ioctl, unsigned long arg)
{
@@ -5906,6 +6168,15 @@ static long kvm_vm_ioctl(struct file *filp,
if (kvm->mm != current->mm || kvm->vm_dead)
return -EIO;
switch (ioctl) {
+ case KVM_SET_PLANE_MEMORY_OVERLAY: {
+ struct kvm_plane_memory_overlay ov;
+
+ r = -EFAULT;
+ if (copy_from_user(&ov, argp, sizeof(ov)))
+ goto out;
+ r = kvm_vm_ioctl_set_plane_overlay(kvm, &ov);
+ break;
+ }
case KVM_CREATE_PLANE:
r = kvm_vm_ioctl_create_plane(kvm, arg);
break;
--
2.50.1