[PATCH v16 06/16] asm-generic: barrier: Add smp_cond_load_acquire_timeout()

[Ankur Arora]

Add the acquire variant of smp_cond_load_relaxed_timeout().

smp_cond_load_acquire_timeout() does its own error checking but
reuses the relaxed logic in __smp_cond_load_relaxed_timeout() for
the actual wait.

Since __smp_cond_load_relaxed_timeout() already provides LOAD->STORE
ordering, we only provide the additional LOAD->LOAD ordering in
the common case.

And providing full ACQUIRE in the error case.

Cc: Kumar Kartikeya Dwivedi <memxor@xxxxxxxxx>
Cc: Alexei Starovoitov <ast@xxxxxxxxxx>
Cc: Arnd Bergmann <arnd@xxxxxxxx>
Cc: Will Deacon <will@xxxxxxxxxx>
Cc: Catalin Marinas <catalin.marinas@xxxxxxx>
Cc: Peter Zijlstra <peterz@xxxxxxxxxxxxx>
Cc: linux-arch@xxxxxxxxxxxxxxx
Cc: bpf@xxxxxxxxxxxxxxx
Reviewed-by: Catalin Marinas <catalin.marinas@xxxxxxx>
Reviewed-by: Haris Okanovic <harisokn@xxxxxxxxxx>
Tested-by: Haris Okanovic <harisokn@xxxxxxxxxx>
Signed-off-by: Ankur Arora <ankur.a.arora@xxxxxxxxxx>
---
Notes:
    With a small change in __smp_cond_load_relaxed_timeout(), it
    now provides LOAD->STORE ordering. (Earlier versions provided
    that only for the non-timeout path.)

    This simplifies providing the ACQUIRE ordering because now we
    don't need special handling for the timeout path.

    For the error path (__scl_timeout_ns < 0) where we just use
    smp_load_acquire().

    In the non-error path we use the control dependency to provide the
    LOAD->STORE ordering with smp_acquire__after_ctrl_dep() providing
    the remaining LOAD->LOAD.

    Unfortunately I couldn't see a way of avoiding the goto label 
    because in this more natural variant the control dependency
    on line 6:

  1	if (unlikely(__scl_timeout_ns < 0)) {
  2		VAL = smp_load_acquire(ptr);
  3	} else {
  4		for (;;) {
  5		   VAL = READ_ONCE(*ptr);
  6		   if (cond_expr || __scl_timed_out)
  7 			break;
  8		   ...
  9		}
 10		smp_acquire__after_ctrl_dep();
 11	}
 12    ...
 

    likely runs afoul of this bit in memory-barriers.txt:
     "control dependencies apply only to the stores in the
      then-clause and else-clause of the if-statement in question
      (including functions invoked by those two clauses), not to
      code following that if-statement."

    IIUC if there was a store on line 12, that would not be ordered
    wrt the control dependency on line 6 and a weakly ordered CPU could
    speculatively execute that earlier especially if the timeout was
    extremely short.
    (An earlier version [1] had a similar problem.)

    So this change disentangles the conditional on line 1 executing
    the if and then clauses sequentially.

[1] https://lore.kernel.org/lkml/20260831202251.305046-7-ankur.a.arora@xxxxxxxxxx/

 include/asm-generic/barrier.h | 47 +++++++++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)

diff --git a/include/asm-generic/barrier.h b/include/asm-generic/barrier.h
index a03e3b26384a..7a59bee1e693 100644
--- a/include/asm-generic/barrier.h
+++ b/include/asm-generic/barrier.h
@@ -383,6 +383,53 @@ do {									\
 })
 #endif
 
+/**
+ * smp_cond_load_acquire_timeout() - (Spin) wait for cond with ACQUIRE ordering
+ * until a timeout expires.
+ * @ptr: pointer to the variable to wait on.
+ * @cond_expr: boolean expression to wait for.
+ * @time_expr_ns: monotonic expression that evaluates to time in ns or,
+ *  on failure, returns a negative value.
+ * @timeout_ns: timeout value in ns. A timeout value of zero is treated
+ *  as the minimum wait so we evaluate cond_expr at least once.
+ * (Both of the above are assumed to be compatible with s64.)
+ *
+ * Equivalent to using smp_cond_load_acquire() on the condition variable with
+ * a timeout.
+ */
+#ifndef smp_cond_load_acquire_timeout
+#define smp_cond_load_acquire_timeout(ptr, cond_expr,			\
+				      time_expr_ns, timeout_ns)		\
+({									\
+	__label__ __scl_out;						\
+	s64 __scl_timeout_ns = (s64)(timeout_ns);			\
+	__unqual_scalar_typeof(*(ptr)) VAL;				\
+									\
+	if (unlikely(__scl_timeout_ns < 0)) {				\
+		VAL = smp_load_acquire(ptr);				\
+		goto __scl_out;						\
+	}								\
+									\
+	VAL = __smp_cond_load_relaxed_timeout(ptr,			\
+					      cond_expr,		\
+					      time_expr_ns,		\
+					      __scl_timeout_ns);	\
+	/*								\
+	 * We return having waited on the control dependency,		\
+	 * (cond_expr || __scl_timed_out). Since we break out only	\
+	 * after either of these conditions are true, the conditional	\
+	 * separates the execution stream into two happened-before	\
+	 * and happened-after sequences. And since STOREs are not	\
+	 * speculated, this takes cares of the LOAD->STORE ordering.	\
+	 *								\
+	 * Provide the remaining LOAD->LOAD ordering.			\
+	 */								\
+	smp_acquire__after_ctrl_dep();					\
+__scl_out:								\
+	(typeof(*(ptr)))VAL;						\
+})
+#endif
+
 /*
  * pmem_wmb() ensures that all stores for which the modification
  * are written to persistent storage by preceding instructions have
-- 
2.43.7