[PATCH] mm/page_alloc: use batch page clearing in kernel_init_pages()

[Hrushikesh Salunke]

When init_on_alloc is enabled, kernel_init_pages() clears every page
one at a time, calling clear_page() per page.  This is unnecessarily
slow for large contiguous allocations (mTHPs, HugeTLB) that dominate
real workloads.

On 64-bit (!HIGHMEM) systems, switch to clearing pages in batch via
clear_pages(), bypassing the per-page kmap_local_page()/kunmap_local()
overhead and allowing the arch clearing primitive to operate on the full
contiguous range in a single invocation.  The batch size is the full
allocation when the preempt model is preemptible (preemption points are
implicit), or PROCESS_PAGES_NON_PREEMPT_BATCH otherwise, with
cond_resched() between batches to limit scheduling latency under
cooperative preemption.

The HIGHMEM path is kept as-is since those pages require kmap.

Allocating 8192 x 2MB HugeTLB pages (16GB) with init_on_alloc=1:

  Before: 0.445s
  After:  0.166s  (-62.7%, 2.68x faster)

Kernel time (sys) reduction per workload with init_on_alloc=1:

  Workload            Before       After       Change
  Graph500 64C128T    30m 41.8s    15m 14.8s   -50.3%
  Graph500 16C32T     15m 56.7s     9m 43.7s   -39.0%
  Pagerank 32T         1m 58.5s     1m 12.8s   -38.5%
  Pagerank 128T        2m 36.3s     1m 40.4s   -35.7%

Signed-off-by: Hrushikesh Salunke <hsalunke@xxxxxxx>
---
base commit: 1a2fbbe3653f0ebb24af9b306a8a968287344a35

 mm/page_alloc.c | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index b1c5430cad4e..178cbebadd50 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1224,8 +1224,23 @@ static void kernel_init_pages(struct page *page, int numpages)
 
 	/* s390's use of memset() could override KASAN redzones. */
 	kasan_disable_current();
-	for (i = 0; i < numpages; i++)
-		clear_highpage_kasan_tagged(page + i);
+
+	if (!IS_ENABLED(CONFIG_HIGHMEM)) {
+		void *addr = kasan_reset_tag(page_address(page));
+		unsigned int unit = preempt_model_preemptible() ?
+					numpages : PROCESS_PAGES_NON_PREEMPT_BATCH;
+		int count;
+
+		for (i = 0; i < numpages; i += count) {
+			cond_resched();
+			count = min_t(int, unit, numpages - i);
+			clear_pages(addr + (i << PAGE_SHIFT), count);
+		}
+	} else {
+		for (i = 0; i < numpages; i++)
+			clear_highpage_kasan_tagged(page + i);
+	}
+
 	kasan_enable_current();
 }
 
-- 
2.43.0