Re: [PATCH 0/4] sched/numa: Add per-process automatic NUMA balancing control

[Yafang Shao]

On Sat, Oct 17, 2026 at 4:32 AM Tim Chen <tim.c.chen@xxxxxxxxxxxxxxx> wrote:
>
> On Fri, 2026-10-16 at 16:18 +0800, Yafang Shao wrote:
> > On Fri, Oct 16, 2026 at 4:01 PM Chen, Yu C <yu.c.chen@xxxxxxxxx> wrote:
> > >
> > > On 10/16/2026 3:38 PM, Yafang Shao wrote:
> > > > On Fri, Oct 16, 2026 at 1:51 PM Chen Yu <yu.c.chen@xxxxxxxxx> wrote:
> > > > > A more complex scenario is when cgroup A and cgroup B become threaded domains:
> > > > >
> > > > >                           root
> > > > >                          /    \
> > > > >                         X      Y
> > > > >                         |      |
> > > > >                         A      B          A, B: threaded domains (thread roots)
> > > > >                        / \    / \         separate memory.max / io.max
> > > > >                      A1   A2 B1  B2       threaded cgroups
> > > > >                      ^        ^
> > > > >                      |        |
> > > > >                      +--------+
> > > > >                    want to share one cache group
> > > > >
> > > > > In the above case, we might need to change the deployed cgroup hierarchy.
> > > >
> > > > I think we can enable LLC-aware scheduling for both A1 and A2 — does
> > > > that sound right?
> > > >
> > >
> > > Here A1 wants to share data with B1, but A2 does not want to share data
> > > with B2 nor B1 nor A1. Enabling cache-aware for A1 and A2 might not help
> > > in this case.
> >
> > I'm not sure whether this is a real use case or merely a theoretical
> > one. I'd suggest we not focus too much on purely theoretical scenarios
> > unless someone can demonstrate a practical need.
>
> In our discussions with Vern Hao from Tencent, they have use cases where
> A1 and B1 are dealing with data compressions while A2 and B2 are dealing
> with databases.  So data are mostly shared between A1 and B1, A2 and B2
> due to functionality.  We may put the parent A or B in the same sched group,
> but the sched group may contain too many threads to fix in a LLC.

Good to know. While I'm not sure why it's deployed that way, I
understand that real-world use cases are often complex.

>
> The tricky thing is if there are scenarios where
> we want to do grouping one way when we consider resource
> usage but another way when we consider data sharing.
>
> >
> > >
> > > > >
> > > > > Suppose threads in A1 and threads in A2 belong to the same process Pa,
> > > > > this reminds me of the cgroup bpf NUMA balancing case: after enabling
> > > > > NUMA balancing for A1 but not setting it for A2, what would be the final
> > > > > NUMA balancing decision for process Pa, is it enabled or disabled?
> > > > > The NUMA balancing scan/invalid seems to be per process, so if we change
> > > > > the setting of its threads, the whole process will be impacted. Or maybe
> > > > > we should not enable NUMA balancing for threaded cgroups? Anyway I haven't
> > > > > digged into your cgroup bpf patch set yet, let me try to look at it.
> > > >
> > > > cgroup NUMAB works the same way: the setting is per cgroup and opt-in.
> > > > For Pa in your example, since its threads spread across A1 and A2, you
> > > > would enable it on both — enabling it on only one of them is not
> > > > meaningful.
> > > >
> > >
> > > I see. I haven't yet digged into your code, is there a per cgroup flag
> > > cgroup.numab.enabled?
> >
> > Right. I will introduce a cgroup::numab_mode field, but without a
> > cgroupfs interface file; it will be written through a BPF program.
> >
> > > Would there be a logic in the kernel, when
> > > A1.numab.enabled
> > > is written to true, the kernel detects all the other cgroups {X} that
> > > contains threads
> > > share the same mm_struct as A1's threads/tasks, and then update {X}'s
> > > numab.enabled
> > > to the same value as A1, to keep consistency?
> >
> > We have no plan to support threaded cgroups for now. If a real need
> > arises, the BPF program can be extended to handle it, which would just
> > require more kfuncs.
> >
> > > In summary, if threads are
> > > spread
> > > cross different cgroups, when you set the enable flag for 1 cgroup, will the
> > > kernel iterates all the other cgroups to update their flags as well? Or
> > > leverage userspace to do that?
> >
> > I believe this should be implemented in userspace, not automatically
> > in the kernel. Use cases are always complex, and flexibility is
> > important here.
>
> I think there is a misunderstanding that using prctl necessarily mean
> we have to modify the user programs.
>
> With the prctl that Chen Yu and I proposed for cache aware scheduling
> [1], the application does not have to change at all. The prctl can be
> issued by an orchestrator daemon on other tasks:
> the caller passes a target pid and a pid_type (thread,
> thread group or process group), and the permission check is
> ptrace_may_access(), as in core scheduling. So a daemon run by the
> platform team can put the application's threads into the right cache
> group from outside:
>
>   prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_CREATE, leader, 0, PIDTYPE_TGID);
>   prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_SHARE_FROM, tid, leader, PIDTYPE_PID);
>
> The daemon can take its policy from wherever is convenient: cgroup
> membership, the pod spec, process names, or a config file. Changing
> the policy means changing the daemon, which the same team owns. That
> is the same round-trip as changing a BPF program, and no application
> team is involved. This is how schedqos already uses the core
> scheduling prctl.
>
> That said, we may start with a per cgroup control of
> cache aware scheduling that sit next to the prctl [1]
> without either one replacing the other.  A lot of the
> code in [1] can be reused with per cgroup control
> of cache aware scheduling.
>
> It starts small, in the form Tejun suggested for NUMA balancing
> in your series [2], and grows only when a real need shows up.

That's great.

>
> The rule for every step with a BPF based cgroup interface:
> the per-task p->sched_cache_grp from [1]
> is the only thing the scheduler reads. Each interface is just
> another way to set it, so the scheduler side never changes.
>
> Step 1: a per-cgroup setting
> ----------------------------
>
>   - struct cgroup gets a sched_cache_grp pointer and a mode:
>     inherit (default), off, or "this subtree is one group".
>   - One kfunc sets the mode by cgroup id. No cgroupfs file for now.
>   - The kernel puts tasks in the cgroup's group:
>       - when the mode is set: allocate the group, publish it, then
>         walk the cgroup's tasks
>       - at fork: in sched_post_fork(), right after cgroup_post_fork()
>       - at exec: exec_mmap() installs the cgroup's group instead of the
>         mm's one
>       - on migration: in cgroup_migrate_execute(), next to the
>         ss->attach() calls
>   - A task that was grouped explicitly (by the prctl, or later by BPF)
>     keeps its group. Cgroup changes do not override it.
>   - In the "advise" mode the cgroup's group is created enabled, since
>     a new group starts disabled there.
>
> This reuses most of [1]: the refcounted sched_cache_group (patches
> 1-3), __sched_cache_set() with the pi_lock swap and the footprint
> transfer (patch 4), the fork and exec handling (patches 2 and 4), the
> per-group enabled flag (patch 5) and the always/advise/never mode
> (patch 6). Only the prctl UAPI itself is not needed.
>
> This covers the the case where the sched groups match well with
> cgroups: cache aware scheduling can be turned on for group A
> only. The other part, a footprint threshold for A that differs from
> the global llc_aggr_tolerance, would need a per-group tolerance that
> the per-cgroup setting can carry.
>
> Step 2: BPF picks the group
> ---------------------------
>
> Add kfuncs to create, read, set and reset a group (thin wrappers around
> the same code), and one that stores a BPF-chosen group in a cgroup.
> The kernel still installs tasks at fork, exec and migration as in step
> 1. The difference is that two cgroups can now share one group, for
> example A1 and B1 in one sched group in the above example.
>
> Step 3: BPF decides per task
> ----------------------------
>
> Only if someone needs it: programs on the fork, exec and cgroup-attach
> hooks that set a task's group directly, for example to split a cgroup
> that is too big for one LLC, or to group only some of its tasks. The
> set kfunc takes pi_lock, so it needs a context filter that allows it
> only in sleepable programs and those hooks.
>
> Each step stands on its own. If step 1 is all anyone ever needs, we
> stop there. The prctl stays for users who do not run cgroups or schedqos.
>
> Does step 1 match what you have in mind for the LLC-aware scheduler
> on your servers?

Thanks for the detailed explanation.  It works for me.

My original idea was a simpler cgroup-field-only scheme: store the
mode on the cgroup and derive each task's group from its cgroup at
lookup time, which would avoid the fork/exec/migration hooks. On a
closer look that does not hold up for LLC: the prctl can set a
task's group explicitly and that has to keep precedence, and the
footprint estimate has to move between groups on a membership
change. Since most of the infrastructure is reused from the
prctl()-based series, I'm fine with implementing it that way.

Please Cc me when you post it.

>
> [1] https://lore.kernel.org/lkml/cover.1787955777.git.tim.c.chen@xxxxxxxxxxxxxxx/
> [2] https://lore.kernel.org/lkml/20261012120438.52849-1-laoar.shao@xxxxxxxxx/
>
> Thanks.
>
> Tim


--
Regards
Yafang