Re: [RFC PATCH 0/6] sched/fair: Introduce the SMT BUDDY ASSIST feature

[Li kunyu]

Hi Chenyu,

On 2026/10/23 0:02, Chen Yu wrote:
> Hi Kunyu,
> 
> On Thu, Oct 22, 2026 at 06:08:21PM +0800, Li kunyu wrote:
>> From: Peter Zijlstra <peterz@xxxxxxxxxxxxx>
>>
>> On Thu, Oct 22, 2026 at 03:40:44PM +0800, Peter Zijlstra wrote:
>>> On Tue, Oct 20, 2026 at 04:37:40PM +0800, Li kunyu wrote:
>>>> [Problem]
>>>>
>>>> It was found that on servers equipped with the Hygon 7447v CPU (Model 7,
>>>> 384 logical cores), migrating tasks across LLC scheduling domains during
>>>> load balancing caused significant performance degradation:
>>>>
>>>> 1. A low L3 cache hitting rate (due to factors such as non-shared L3
>>>> caches).
>>>>
>>>> 2. The traversal overhead of higher-level scheduling domains (e.g.,
>>>> cross-NUMA scheduling domains).
>>>>
>>>> Similar issues were also observed on CPUs from other vendors during
>>>> testing, though the degree of memory bandwidth increase and other
>>>> latency effects varied. Consequently, a series of patches was developed
>>>> to address and improve these issues.
>>>>
>>>>
>>>> [Solution]
>>>>
>>>> This RFC patch series introduces the SMT_BUDDY_ASSIST scheduling
>>>> feature. The main work is as follows:
>>>>
>>>> 1. Push CFS tasks to SMT sibling CPUs during periodic load balancing.
>>>>
>>>> 2. Skip SMT-domain balancing after a successful push and avoid pulling
>>>> the pushed task back.
>>>>
>>>> 3. Restrict newidle balancing so it does not pull from higher-level
>>>> scheduling domains.
>>>>
>>>> 4. Compensate for the resulting load imbalance with a burst-task wakeup
>>>> path that sends tasks to idle CPUs.
>>> So you're combatting undue LLC migrations with brute force :-( Mostly
>>> this looks like something where cache aware scheduling should help, no?
>>> That ought to make cross-LLC migrations less attractive.
>>>
>>> Now, I realize that cache-aware scheduling as is won't help stream; when
>>> build using OpenMP the whole thing is one big process and cache-aware
>>> mostly gives up, when build using MPI its a ton of single threads and
>>> cache-aware will ignore them.
>>>
>>> Still, I think you really should look that way, perhaps investigate if
>>> something like:
>>>
>>>   https://patch.msgid.link/cover.1787955777.git.tim.c.chen%40linux.intel.com
>>>
>>> can help you.
>> Thank you for your suggestion. I looked at this patch series and think
>> the way it exposes the interface to users ("always" | "advise" |
>> "never") is quite good. We will add the related interfaces in a subsequent
>> version.
>>
>>
> If I understand correctly, the goal of your patch set is to inhibit cross-node
> task migration so as to keep memory locality.
> 
> Patch 4 is the main one, which forbids the newly-idle cross-node balancing. As
> a result of Patch 4, more tasks end up on a single node, so the kernel needs to
> spread them within that node more aggressively, this is why Patches 1, 2, 3 and
> 5 were introduced.
> 
> Before digging further into the design, I think one question needs to be clarified:
> In the scenario you are trying to optimize, do the workload's threads share data
> with each other?
> 
> 1. If the threads share data with each other, then leveraging cache-aware scheduling
>    with prctl to put them together in the same LLC/node might be a good choice.
>    Furthermore, there is a proposal [1] to enhance prctl to allow the user to specify
>    how aggressively the threads should be aggregated, which would work in the "stream"
>    case.

First, this series is intended to address both scenarios. It is designed to be
generic, requiring no manual configuration from users, and it delivers solid
improvements in both cases. Let me start with Scenario 1.

prctl gives users a clear view of which CPUs and which process/thread ranges
(for example, those typically running in cgroups) are restricted. It can also be
combined with this patch series to achieve better performance. For instance, in
patch 5, nohz.idle_cpus_mask can be used together with the prctl-restricted CPU
mask to select idle CPUs within the prctl-restricted range, which can further
improve local performance.

These patches and CAS are complementary: each improves its own specialization
for specific scenarios. I therefore believe this patch series is worthwhile.

> 2. If the threads do not share data with each other and have a high memory footprint,
>    then leaving them where their memory resides would be a good choice. This is what
>    NUMA balancing tries to achieve, and it is possible that threads of the same process
>    are spread on different nodes.

Regarding the second scenario, this patch series has a decisive advantage. Patch
4 delivers an overall average improvement of 3.28%, while Patch 5 delivers an
overall average improvement of more than 13%. In the patch tests I am currently
optimizing, these figures are still increasing - the overall average improvement
is now over 18%, which is hard to believe, so I will continue to verify it
several more times. Some cases that previously showed performance degradation
have now fallen to even lower values.

Patches 1, 2, and 3 have been revised to have more reasonable semantics, as
their goal is to distribute all CPU tasks more efficiently within a shorter
period of time. Of course, they can also be used together with restrictions such
as cgroups.

This patch series achieves an average improvement of over 50% across the four
test categories in stream-10 and stream-100, giving it a decisive advantage in
throughput.

> From the [Problem] section in the cover letter, it seems you are trying to address case 2?
> If this is the case, why doesn't NUMA balancing help to inhibit the cross-node task migration?
> Or is it because NUMA balancing is disabled on your platform?
> 
> Besides, there is a patch set to extend cache-aware scheduling to cover multiple
> preferred LLCs and nodes [2], so as to mitigate cross-LLC/cross-node latency as much as
> possible, not sure whether this work addresses the issue you mentioned.
> 
> [1] https://lore.kernel.org/lkml/tencent_93116D14C771DC8C988C10E3C634BE0CC107@xxxxxx/
> [2] https://lore.kernel.org/lkml/20260827122816.756234-1-wujianyong@xxxxxxxx/

I have read both patch series. Series [1] is more in line with Scenario 1
mentioned above, whereas series 2 is an extension of a particular scenario in
the CAS part. Neither can fully solve the problems I described. However, these
two series and this patch series are complementary, and each provides
performance improvements in its respective area.

-- 
thanks,
kunyu