Re: [linus:master] [mm/rmap] 6af8cb80d3: vm-scalability.throughput 7.8% regression

[David Hildenbrand]

On 16.04.25 09:01, kernel test robot wrote:
> Hello,
>
> kernel test robot noticed a 7.8% regression of vm-scalability.throughput on:
>
>
> commit: 6af8cb80d3a9a6bbd521d8a7c949b4eafb7dba5d ("mm/rmap: basic MM owner tracking for large folios (!hugetlb)")
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
>
>
> testcase: vm-scalability
> config: x86_64-rhel-9.4
> compiler: gcc-12
> test machine: 256 threads 2 sockets GENUINE INTEL(R) XEON(R) (Sierra Forest) with 128G memory
> parameters:
>
> 	runtime: 300s
> 	size: 8T
> 	test: anon-cow-seq
> 	cpufreq_governor: performance

This should be the scenario with THP enabled. At first, I thought the 
problem would be contention on the per-folio spinlock, but what makes me 
scratch my head is the following:

     13401           -16.5%      11190        proc-vmstat.thp_fault_alloc
...   3430623           -16.5%    2864565        proc-vmstat.thp_split_pmd


If we allocate less THP, performance of the benchmark will obviously be 
worse with less THPs.

We allocated 2211 less THPs and had 566058 less THP PMD->PTE remappings.

566058 / 2211 =  256, which is exactly the number of threads -> 
vm-scalability fork'ed child processes.

So it was in fact the benchmark that was effectively using 16.5% less THPs.

I don't see how this patch would affect the allocation of THPs in any 
way (and I don't think it does).


Regarding possible contention on the spinlock, I was already expecting a 
slight hit once we have that many threads over multiple sockets. From my 
cover letter:

"
Similarly, running these benchmarks with 2 MiB THPs enabled on the
AmpereOne A192-32X with 192 cores, I got < 1% difference with < 1% 
stdev, which is nice.

So far, I did not get my hands on a similarly large system with multiple
sockets.
"

And further:

"
If it ever becomes a problem we could either investigate improving the
locking, or simply stopping the MM tracking once there are "too many
mappings" and simply assume that the folio is "mapped shared" until it
was freed.

[...] Adding that logic to stop adds more code to the hot path, so I 
avoided that for now.
"

So while I am planning on looking into optimizing the locking at some 
point, it has low priority for me because (a) it adds more complexity 
(b) has the potential to affect the hot path (not shared) and (c) this 
benchmark in that scale is not a compelling argument.

--
Cheers,

David / dhildenb