[RFC PATCH 0/2] Optimize S2 page splitting

[Leonardo Bras]

While playing with dirty-bit tracking, I decided to take a look on how page
splitting works. Found out all entries are walked, even though we can infer,
for instance that:
- If a level-3 entry is walked, it means the parent level-2 entry is split
- If a split just succeeded in an table entry, it means all children nodes
  are already split

So I tried to optimize it in a way that it does not break other users.

My main idea is to introduce positive return values that hint to the
pagetable walking mechanism that either siblings or children can be 
skipped. That should be contained to the visitor function, that returns
zero if no error was detected.

Numbers on above optimization are promising:
A 1GB VM, running on the model, splitting all at the beginning 
(no manual protect):
- Memory was already split (4k pages): 	-97.33% runtime (-172ms) - 20 runs
- THP backed memory: 			-19.82% runtime (-153ms) - 10 runs
- 1x1GB hugetlb memory:			-20.65% runtime (-150ms) - 10 runs

This is measured with this snippet[1].
I ran at least 10 times on different 1GB VMs, to make sure the numbers are 
consistent.

Ideas I considered:
- Using a negative return value, using kvm_pgtable_walk_continue to 
  filter it as a non-error, but decided that is kind of counter-intuitive
- Using the introduced return values to hint the split walker to not 
  splitting level-2 blocks (or level-1), if by adding a new parameter in
  kvm_pgtable_stage2_split() and carrying it over to the walker using 
  ctx->arg. (Splitting only up to given hugepage size)
- Looking at other walkers, and trying to think on scenarios to
  optimize them using the new return values.

Do you think it is worth doing this?
Please provide feedback!

Thanks!
Leo


[1]:
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index d089c107d9b7..6424e833b7be 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -1272,22 +1273,26 @@ static void kvm_mmu_split_memory_region(struct kvm *kvm, int slot)
        phys_addr_t start, end;
 
        lockdep_assert_held(&kvm->slots_lock);
 
        slots = kvm_memslots(kvm);
        memslot = id_to_memslot(slots, slot);
 
        start = memslot->base_gfn << PAGE_SHIFT;
        end = (memslot->base_gfn + memslot->npages) << PAGE_SHIFT;
 
+
        write_lock(&kvm->mmu_lock);
+       u64 sw = ktime_get_real_ns();
        kvm_mmu_split_huge_pages(kvm, start, end);
+       sw = ktime_get_real_ns() - sw;
+       printk("split from %llx to %llx took %llu ns\n", start, end, sw);
        write_unlock(&kvm->mmu_lock);
 }
 

Leonardo Bras (2):
  KVM: arm64: Introduce S2 walker SKIP return options
  KVM: arm64: Improve splitting performance by using SKIP return values

 arch/arm64/kvm/hyp/pgtable.c | 32 +++++++++++++++++++++++++-------
 1 file changed, 25 insertions(+), 7 deletions(-)


base-commit: 5d6919055dec134de3c40167a490f33c74c12581
-- 
2.54.0