Re: [RFC PATCH] mm: vmscan: avoid over-reclaim in classic LRU direct reclaim path

[Barry Song]

On Thu, Oct 8, 2026 at 8:01 PM Bo Zhang <zhangbo0325@xxxxxxxxx> wrote:
>
> From: Bo Zhang <zhangbo56@xxxxxxxxxx>
>
> On production Android devices, direct reclaim is a frequent source of
> unpredictable, large latency spikes. A direct reclaim request typically
> asks for only nr_to_reclaim = SWAP_CLUSTER_MAX (32) pages, and in the
> common case completes in well under a millisecond. But the classic LRU
> path has no reliable "goal met" check, so under memory pressure a single
> reclaim can scan tens of thousands of pages and free hundreds to
> thousands - far beyond the 32 requested - before returning.
>
> Pooling the 20 slowest direct-reclaim events, before this patch the
> worst case scanned 130,079 pages and reclaimed 1,635, and the average
> scanned 37,955 pages and reclaimed 881.
>
> The root cause is over-reclaim: the classic LRU path keeps scanning well
> past its target in two ways:
>
> 1. shrink_lruvec() does not stop as soon as nr_to_reclaim is met; it
>    keeps scanning the remaining LRU batches, over-reclaiming past the
>    target.
>
> 2. shrink_node_memcgs() only breaks early on a partial walk. When the
>    initial partial walk misses nr_to_reclaim, do_try_to_free_pages()
>    retries with a full walk, which has no "goal met" check and shrinks
>    every memcg in the tree even after the target is satisfied.
>
> Add an early exit to shrink_lruvec() and shrink_node_memcgs(), and the
> matching check to should_continue_reclaim(), all keyed on
>
>     sc->nr_reclaimed >= max(sc->nr_to_reclaim, compact_gap(sc->order))
>
> The compact_gap() term keeps correctness for high-order reclaim, where
> nr_to_reclaim alone may be too small to free a block of the requested
> order; MGLRU's should_abort_scan() uses the same threshold. Using it in
> all three places keeps the retry decision consistent with where reclaim
> actually stops.
>
> After this patch, over the same 20 slowest events the worst case scans
> 45,593 pages and reclaims 57, and the average scans 6,399 pages and
> reclaims 38 - reclaim now stays close to the requested target instead
> of running far past it. Removing this over-reclaim greatly reduces the
> direct-reclaim tail latency, which significantly cuts the jank caused by
> direct reclaim on interactive workloads.
>
> Signed-off-by: Bo Zhang <zhangbo56@xxxxxxxxxx>
> ---
>  mm/vmscan.c | 35 +++++++++++++++++++++++++++++++++--
>  1 file changed, 33 insertions(+), 2 deletions(-)
>
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 91295070ca33..b1efbd2a9bb6 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -6236,6 +6236,18 @@ static void shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)
>
>                 cond_resched_tasks_rcu_qs();
>
> +               /*
> +                * For high-order allocations, nr_to_reclaim may be smaller
> +                * than what is needed to make a block of the requested order
> +                * available for compaction. When memory is plentiful, keep
> +                * reclaiming until at least compact_gap() pages have been
> +                * freed so the target order can actually be satisfied.
> +                */
> +               if (!cgroup_reclaim(sc) &&
> +                   sc->nr_reclaimed + nr_reclaimed >=
> +                   max(nr_to_reclaim, compact_gap(sc->order)))
> +                       break;
> +

Let's assume `nr[file] = 300` and `nr[anon] = 100` for now.
Breaking out of the loop earlier would break the proportional
scanning of file and anon.

The current algorithm is:

for_each_evictable_lru(lru) {
        if (nr[lru]) {
                nr_to_scan = min(nr[lru], SWAP_CLUSTER_MAX);
                nr[lru] -= nr_to_scan;

                nr_reclaimed += shrink_list(lru, nr_to_scan,
                                            lruvec, sc);
        }
}

This results in the following scan pattern:

32 file - 32 anon
32 file - 32 anon
32 file - 32 anon
32 file -  4 anon
32 file -  0 anon
32 file -  0 anon
32 file -  0 anon
32 file -  0 anon
...

Eventually, the proportional scanning balance is still correct.

With your earlier exit condition, however, we could end up with:

32 file - 32 anon
32 file - 32 anon
32 file - 32 anon
-> target reached, exit

In this case, we scan file and anon at a 1:1 ratio, which
doesn't reflect the intended 3:1 ratio.

The original code maintains the scan ratio by setting the
smaller count to 0 while keeping a nonzero count for the
larger one, allowing it to continue scanning:

              /* Stop scanning the smaller of the LRU */
                nr[lru] = 0;
                nr[lru + LRU_ACTIVE] = 0;

                /*
                 * Recalculate the other LRU scan count based on its original
                 * scan target and the percentage scanning already complete
                 */
                lru = (lru == LRU_FILE) ? LRU_BASE : LRU_FILE;
                nr_scanned = targets[lru] - nr[lru];
                nr[lru] = targets[lru] * (100 - percentage) / 100;
                nr[lru] -= min(nr[lru], nr_scanned);

                lru += LRU_ACTIVE;
                nr_scanned = targets[lru] - nr[lru];
                nr[lru] = targets[lru] * (100 - percentage) / 100;
                nr[lru] -= min(nr[lru], nr_scanned);

This simply reflects the need to respect swappiness.

>                 if (nr_reclaimed < nr_to_reclaim || proportional_reclaim)
>                         continue;
>
> @@ -6345,6 +6357,16 @@ static inline bool should_continue_reclaim(struct pglist_data *pgdat,
>         if (!nr_reclaimed)
>                 return false;
>
> +       /*
> +        * Stop once the overall reclaim goal is met, using the same
> +        * max(nr_to_reclaim, compact_gap()) threshold as the early exits
> +        * in shrink_lruvec() and shrink_node_memcgs(), so the retry
> +        * decision here stays consistent with where reclaim actually
> +        * stops.
> +        */
> +       if (sc->nr_reclaimed >= max(sc->nr_to_reclaim, compact_gap(sc->order)))
> +               return false;
> +
>         /* If compaction would go ahead or the allocation would succeed, stop */
>         for_each_managed_zone_pgdat(zone, pgdat, z, sc->reclaim_idx) {
>                 unsigned long watermark = min_wmark_pages(zone);
> @@ -6442,8 +6464,17 @@ static void shrink_node_memcgs(pg_data_t *pgdat, struct scan_control *sc)
>                                    sc->nr_scanned - scanned,
>                                    sc->nr_reclaimed - reclaimed);
>
> -               /* If partial walks are allowed, bail once goal is reached */
> -               if (partial && sc->nr_reclaimed >= sc->nr_to_reclaim) {
> +               /*
> +                * Break loop once the overall reclaim goal is met. For
> +                * high-order allocations nr_to_reclaim may be too small to
> +                * free a block of the requested order, so when memory is
> +                * plentiful keep going until at least compact_gap() pages
> +                * have been reclaimed. This applies to both partial and full
> +                * walks: a full walk is only a means to reach the goal, not a
> +                * requirement to visit every memcg.
> +                */
> +               if (sc->nr_reclaimed >= max(sc->nr_to_reclaim,
> +                                           compact_gap(sc->order))) {

Sashiko raised a concern that this might break memcg balancing
for kswapd. For example, we might keep reclaiming memory from
the root memcg. This seems to be a valid concern, so perhaps
this should only apply to direct reclaim, not kswapd.


>                         mem_cgroup_iter_break(target_memcg, memcg);
>                         break;
>                 }

Thanks
Barry