[PATCH] mm/filemap: queue non-exclusive folio waiters ahead of exclusive

[M. Grove]

A folio waiter queued behind an exclusive folio_lock() waiter on the same
folio is skipped by folio_wake_bit()'s nr_exclusive=1 walk: a shared/DROP
sleeper (folio_put_wait_locked from do_read_cache_folio) never receives
WQ_FLAG_WOKEN even though its wake ran.  Readers then hang D-state in
folio_wait_bit_common while fresh reads of the same page answer instantly,
and the sleeper wakes only with each later lock/unlock of that address, one
page at a time.

Same symptom as syzbot's "INFO: task hung in read_cache_folio (6)"
(extid 9db0864859224b833108).

The change only reorders the enqueue side: non-exclusive waiters use wait.h's
default head-side insert, exclusive folio_lock waiters keep the tail, so
wake-one fairness for lockers is unchanged.  Built on mm-stable
183895979c44; mm/filemap.o also builds on v7.2.6.

Queue-walk instrument (read-only, counts non-exclusive entries left
unvisited when folio_wake_bit() returns): 363 skipped waiters per 10-minute
run before the change, 1 after the folio_wait_bit_common site, 0 after the
softleaf_entry_wait_on_locked site as well.  Reproducer shape: threads pread
a file while others block in folio_lock on the same pages (hole punch,
truncate or mmap faults) under reclaim churn.  Harness, traces or configs on
request.

Signed-off-by: M. Grove <mgrove@xxxxxxxxxx>
---
 mm/filemap.c | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/mm/filemap.c b/mm/filemap.c
index b74bc1e5015c..35869d19d17a 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -1293,8 +1293,18 @@ static inline int folio_wait_bit_common(struct folio *folio, int bit_nr,
 	 */
 	spin_lock_irq(&q->lock);
 	folio_set_waiters(folio);
-	if (!folio_trylock_flag(folio, bit_nr, wait))
-		__add_wait_queue_entry_tail(q, wait);
+	if (!folio_trylock_flag(folio, bit_nr, wait)) {
+		/*
+		 * A non-exclusive (shared/DROP) waiter must queue ahead of
+		 * exclusive folio_lock waiters: the nr_exclusive=1 wake walk
+		 * stops at the first exclusive match, so a non-exclusive
+		 * waiter behind one is never visited and never woken.
+		 */
+		if (behavior == EXCLUSIVE)
+			__add_wait_queue_entry_tail(q, wait);
+		else
+			__add_wait_queue(q, wait);
+	}
 	spin_unlock_irq(&q->lock);
 
 	/*
@@ -1428,7 +1438,7 @@ void softleaf_entry_wait_on_locked(softleaf_t entry, spinlock_t *ptl)
 	spin_lock_irq(&q->lock);
 	folio_set_waiters(folio);
 	if (!folio_trylock_flag(folio, PG_locked, wait))
-		__add_wait_queue_entry_tail(q, wait);
+		__add_wait_queue(q, wait);
 	spin_unlock_irq(&q->lock);
 
 	/*
-- 
2.56.0