Re: [PATCH] mm: nommu: set MMF_OOM_SKIP in exit_mmap()

[Michal Hocko]

On Fri 16-10-26 00:42:39, Alexis Bouzigues wrote:
> On Thu, Oct 15, 2026 at 09:03:57AM +0200, Michal Hocko wrote:
> > Are you really looking into the code or just relying on an LLM?
> > Go and check to learn that the oom reaper is only compiled in with
> > CONFIG_MMU=y
> 
> Hello,
> 
> Sorry about that, my earlier message used "reap" in the sense of
> cleaning up the exited child in zombie state (waitpid), not in the sense
> of oom-reaper.

OK, this is what confused me indeed. Especially along with the changelog
explicitly mentioning coordination with the oom reaper
: Fixes: 212925802454 ("mm: oom: let oom_reap_task and exit_mmap run concurrently")

The whole oom reaper chnages should have no impact on the nommu OOM
workflow.

> I also went through the code in detail today manually using a 
> backtrace from the reproducer. I set a breakpoint at oom_evaluate_task 
> (containing the MMF_OOM_SKIP check) ran the reproducer on v7.3-rc7
> through qemu-system-arm with gdb. I iterated to confirm the
> breakpoint keeps getting hit.
> 
> Here is a backtrace of the reproducer when the parent PID is livelocked:
> 
>     oom_evaluate_task                  mm/oom_kill.c:354
>     select_bad_process                 mm/oom_kill.c:373
>     out_of_memory                      mm/oom_kill.c:1155
>     __alloc_pages_may_oom              mm/page_alloc.c:4113
>     __alloc_pages_slowpath             mm/page_alloc.c:5030
>     __alloc_frozen_pages_noprof        mm/page_alloc.c:5449
>     __alloc_pages_noprof               mm/page_alloc.c:5471
>     alloc_pages_node_noprof            mm/page_alloc.c:5485
>     alloc_pages_noprof                 include/linux/gfp.h:284
>     get_free_pages_noprof              mm/page_alloc.c:5507
>     alloc_pages_exact_noprof           mm/page_alloc.c:5641
>     do_mmap_private                    mm/nommu.c:959
>     do_mmap                            mm/nommu.c:1182
>     vm_mmap_pgoff                      mm/util.c:581
>     ksys_mmap_pgoff                    mm/nommu.c:1287
> 
> 
> Inside oom_evaluate_task(), when the task is the zombie child pid
> victim, without MMF_OOM_SKIP it aborts the loop from select_bad_process()
> and sets oc->chosen to -1, preventing the oom killer from calling
> oom_kill_process on oc->chosen

Correc.

> Since oc->chosen is not NULL, it does not hit the deadlock branch in
> out_of_memory instead it returns !!oc->chosen (1).
> 
> Returning to __alloc_pages_may_oom(), out_of_memory(&oc) evaluates
> non-zero, so did_some_progress from __alloc_pages_slowpath gets set
> to 1.
> 
> Then in __alloc_pages_slowpath(), since did_some_progress was set
> to 1, it jumps to the retry label
> 
> As long as the system is out of memory, __alloc_pages_may_oom and 
> out_of_memory will get called, meaning that oom killer does not make
> progress and that the parent PID busy spins in the allocation.

Correct. The question is why killing the child, which has passed
exit_mmap stage sucessfully didn't help.

> If the MMF_OOM_SKIP bit were to be set for the child zombie process, then
> oom_evaluate_task would not abort and would select the parent, meaning the
> oom killer can now call oom_kill_process on oc->chosen and progress.
> 
> So setting MMF_OOM_SKIP as soon as its memory gets freed (in exit_mmap)
> would guard against the parent's mmap stalling oom killer, since
> oom_evaluate_task would rule the child out and proceed.

right MMF_OOM_SKIP will help to hide the killed child process. The
question is whether this is just working around a propblem or it is a
correct solution/workaround. There is absolutely no guarantee of a
forward progress for nommu (due to missing oom_reaper) and overall
nommu OOM handling is rather fragile and there is an implicit assumption
that memory usage in this platform is very carefully scrutinized to make
sure this doesn't really happen.

So the main question you should focus on to explain is whether your
specific example is  
	a) a corner case of a valid nommu application or  this is 
	   a theoretical problem
	b) why this doesn't break otherwise very fragile nommu oom
	   handling
Thanks!
-- 
Michal Hocko
SUSE Labs