Re: [PATCH 8/8] RFC: use a TASK_FIFO kthread for read completion support

[David Laight]

On Tue, 14 Apr 2026 10:58:16 +1000
Dave Chinner <dgc@xxxxxxxxxx> wrote:

> On Sat, Apr 11, 2026 at 07:44:43AM +0800, Gao Xiang wrote:
> > 
> > 
> > On 2026/4/11 06:11, Dave Chinner wrote:  
> > > On Thu, Apr 09, 2026 at 06:02:21PM +0200, Christoph Hellwig wrote:  
> > > > Commit 3fffb589b9a6 ("erofs: add per-cpu threads for decompression as an
> > > > option") explains why workqueue aren't great for low-latency completion
> > > > handling.  Switch to a per-cpu kthread to handle it instead.  This code
> > > > is based on the erofs code in the above commit, but further simplified
> > > > by directly using a kthread instead of a kthread_work.
> > > > 
> > > > Signed-off-by: Christoph Hellwig <hch@xxxxxx>  
> > > 
> > > Can we please not go back to the (bad) old days of individual
> > > subsystems needing their own set of per-cpu kernel tasks just
> > > sitting around idle most of of the time?  The whole point of the
> > > workqueue infrastructure was to get rid of this widely repeated
> > > anti-pattern.
> > > 
> > > If there's a latency problem with workqueue scheduling, then we
> > > should be fixing that problem rather than working around it in every
> > > subsystem that thinkgs it has a workqueue scheduling latency
> > > issue...  
> > 
> > It has been "fixed" but never actually get fixed:
> > https://lore.kernel.org/r/CAB=BE-QaNBn1cVK6c7LM2cLpH_Ck_9SYw-YDYEnNrtwfoyu81Q@xxxxxxxxxxxxxx
> > 
> > and workqueues don't have any plan to introduce RT threads;  
> 
> They don't need to (or should) introduce RT threads. Per-cpu kernel
> threads already get priority over normal user tasks on scheduling
> decisions.  However, they do not pre-empt running kernel tasks of
> the same priority.
> 
> In general, kernel threads should not use RT scheduling at all - if
> the kernel uses RT prioprity tasks then that can interfere with user
> scheduled RT tasks. This is especially true in this case where a
> non-RT tasks issue the IO, and the IO completion is then scheduled
> with RT priority. IOWs, any unprivileged user can now impact the
> processing time available to, and the response latency of, other
> RT scheduled tasks the system is running.

But might not an IO for a use RT task be sat behind it in the
completion queue?
To avoid priority inversion you need to process the completions.
The non-RT task won't be rescheduled to issue a later request.
The only other way is to force that completion work be done in
the context of the thread that issued the request.

> 
> Tejun asked Sandeep if setting the workqueue thread priority to
> -19 through sysfs (i.e. making them higher priority than normal
> kernel threads) had the same effect on latency as using a dedicated
> per-cpu RT task thread. THere was no followup.
> 
> In theory, this should provide the same benefit, because what RT
> scheduling is doing is pre-empting any user and kernel task that was
> running when the interrupt was delivered to execute the completion
> task immediately.
> 
> Setting the workqueue to use kernel threads of a higher scheduler
> prioirty should do the same thing, without the need to use dedicated
> per-cpu RT threads.

We had a related issue with the network stack's use of softint and
threaded napi when trying to receive very high packet rate UDP traffic.
(IIRC something like 500000 200byte RTP audio packets every second.)
There is an absolute requirement to run the ethernet rx processing
in order to avoid dropping packets.
Now normally the code runs as 'softint' making it higher priority
that any user process, but under load it switches to using 'normal
priority' kernel threads which are basically low priority.
Similarly 'threaded napi' uses 'normal priority' threads.
The only we made it work was to manually change the threads to be
a low 'RT FIFO' priority so they got scheduled in preference to
all user processes.
A system may need some user RT threads with a lower priority than
these kernel threads and others with a higher priority.
So defaulting to a middling RT priority may be best.

	David