Re: [PATCH v2] mm/khugepaged: clear MMF_VM_HUGEPAGE on mm_slot_alloc() failure

[David Hildenbrand (Arm)]

On 5/6/26 12:51, Lance Yang wrote:
> 
> On Wed, May 06, 2026 at 12:16:35PM +0530, Dev Jain wrote:
>>
>>
>> On 06/05/26 6:51 am, Ye Liu wrote:
>>> From: Ye Liu <liuye@xxxxxxxxxx>
>>>
>>> __khugepaged_enter() sets MMF_VM_HUGEPAGE before allocating the
>>> corresponding mm_slot.  If mm_slot_alloc() fails, the function
>>> returns with the flag set but without inserting the mm into the
>>> khugepaged tracking structures.
>>>
>>> This leaves the mm in an inconsistent state: it is marked as
>>> registered (MMF_VM_HUGEPAGE set), but will never be scanned by
>>> khugepaged.  Future attempts to register the mm are skipped since
>>> khugepaged_enter_vma() checks the flag and returns early.
>>>
>>> Fix this by clearing MMF_VM_HUGEPAGE when mm_slot_alloc() fails,
>>> restoring the ability to retry registration later.
>>>
>>> Fixes: 16618670276a ("mm: khugepaged: avoid pointless allocation for struct mm_slot")
>>> Signed-off-by: Ye Liu <liuye@xxxxxxxxxx>
>>> ---
>>> Changes since v1:
>>> - Add Fixes tag as suggested by Dev Jain and Lance Yang
>>>
>>>  mm/khugepaged.c | 4 +++-
>>>  1 file changed, 3 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/mm/khugepaged.c b/mm/khugepaged.c
>>> index 7d48d4fbd5f3..60ab7c1b61dd 100644
>>> --- a/mm/khugepaged.c
>>> +++ b/mm/khugepaged.c
>>> @@ -559,8 +559,10 @@ void __khugepaged_enter(struct mm_struct *mm)
>>>  		return;
>>>  
>>>  	slot = mm_slot_alloc(mm_slot_cache);
>>> -	if (!slot)
>>> +	if (!slot) {
>>> +		mm_flags_clear(MMF_VM_HUGEPAGE, mm);
>>>  		return;
>>> +	}
>>
>> Note that, a racing khugepaged_enter_vma() may back off
>> when it sees that MMF_VM_HUGEPAGE is set, but then the above
>> clears the flag after slot alloc failure. So we end up not
>> registering the mm with khugepaged. But I am sure no one
>> cares, we are in much big trouble if slot alloc is failing.
> 
> Right. A racing khugepaged_enter_vma() can see MMF_VM_HUGEPAGE is set
> and return, then !slot clears it again. If there is no later
> khugepaged_enter_vma(), the mm still wouldn't get registered :)

So why not

diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 5f4e009593e0..78735f34250a 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -437,13 +437,16 @@ void __khugepaged_enter(struct mm_struct *mm)

        /* __khugepaged_exit() must not run from under us */
        VM_BUG_ON_MM(collapse_test_exit(mm), mm);
-       if (unlikely(mm_flags_test_and_set(MMF_VM_HUGEPAGE, mm)))
-               return;

        slot = mm_slot_alloc(mm_slot_cache);
        if (!slot)
                return;

+       if (unlikely(mm_flags_test_and_set(MMF_VM_HUGEPAGE, mm))) {
+               mm_slot_free(mm_slot_cache, slot);
+               return;
+       }
+
        spin_lock(&khugepaged_mm_lock);
        mm_slot_insert(mm_slots_hash, mm, slot);
        /*


Arguably, on the race described above, likely the thread seeing the
MMF_VM_HUGEPAGE would likely similarly have failed the allocation.

I'm fine with either, just wanted to raise the (cleaner looking?) alternative
where we just properly back off?

-- 
Cheers,

David