Re: [RFC PATCH 0/3] Neural Storage Driver - learning page cache prefetcher

[ayhan aydın]

Hi Honza, hi Willy,

First, my Aug 20 follow-up bounced on the HTML part. My mistake, resending
as plain text. Below is that content updated with six more weeks of
measurement. Two self corrections first, before any numbers.

What changed since the RFC (same out of tree module):
Automated prefetch never exceeds ra_pages now. Overhangs get trimmed
or dropped, all counted. The only exception is our own measured
willneed ranges up to io_pages. That follows Ming Lei's RFC direction
and the MADV_POPULATE precedent. It is toggleable, Mod A means strict cap.

Fault path first, as you asked in Q3. filemap_fault hook observes, then
prefetches with depth capped at 2 and bounded by the cap. One more
thing on this: the fault path stands down explicitly at 1M windows and
up. There is a straight s_bdi->ra_pages check in nsd_process(). The
  parity we see at 256K and 512K is a different thing. Our code still
  fires there. The kernel just catches up on its own at those sizes.

A sleeping in atomic bug in our fast path got found through a stack
trace and fixed. vfs_fadvise was called from fprobe context. It now
goes through a workqueue.

A len==0 fadvise footgun got found through our own counters. Len 0
means to end of file, so those sends preloaded whole files. All our
earlier big mmap and SQLite speedups traced back to it. Retracted.
Sends with len 0 are now dropped and counted.

Narrowed claims after all that. Interleaved A/B, drop_caches before
every run, checksums verified, 500+ runs in total:

  sparse mmap scan, about 64KB stride, 512MB to 20GB files, 128K window,
  SSD: ON ~47s vs OFF ~88s median over 30+30 reps, no overlap.
    The threshold lines up exactly with FAULT_AROUND_ORDER=4 (65536B).
    Below it the kernel fault around covers the fault by itself.
    Above it, it does not.
  same setup at 256K, 512K, 1024K windows: parity, kernel catches up.
  dense mmap, rand 4K, seq read, SQLite, THP, and HDD sparse mmap:
  parity everywhere. The early small HDD lean got remeasured with
  current code and traced to the same len 0 accident, retracted too.
  No workload regresses. dmesg tracked throughout, no new warnings.

So the scope is sparse mmap scans at default size windows on
SSD. That is where the fault path underprovisions while the read path
is already saturated. Dense, random, 1M tuned and HDD cases are parity
by measurement.
Separately, one thing that might be useful here too: p99 tail latency
is about 4x worse at 1M than at 128K regardless of NSD state. Another
practical reason to keep windows small.

Open questions, updated:
  Q1, fadvise beyond ra_pages: we stopped doing it silently. Only
      measured willneed ranges up to io_pages, and that can be turned
      off. Acceptable shape?
  Q2, ra_state integration: we prototyped window steering from the
      module. The kernel keeps our window but wall clock does not move
      because there is just not enough volume from outside. Real
      integration needs in kernel changes, so we drafted a minimal one:
      2 stream slots plus stride state in file_ra_state, every window
      bounded by ra_pages, read path only. Want me to post it separately
      for direction feedback before any boot testing?
  Q3: fault path first as described above, with timely versus correct
      accounting running underneath.

For Willy: backward patterns get detected near perfectly (strides with
negative deltas, counted separately) but they do not move the wall on
SATA SSD, so throttled now. Fault path is covered as you asked.
PG_readahead replacement stays an in kernel question as above.

Full data and the retraction notes are in the tree. Happy to run any
counter matrix you ask for.

Thanks,
Ayhan