Re: How to handle P2P DMA with only {physaddr,len} in bio_vec?

[Christoph Hellwig]

On Tue, Jun 24, 2025 at 10:02:05AM +0100, David Howells wrote:
> > There isn't a very easy way.  Also because if you actually need to do
> > peer to peer transfers, you right now absolutely need the page to find
> > the pgmap that has the information on how to perform the peer to peer
> > transfer.
> 
> Are you expecting P2P to become particularly common?

What do you mean with 'particularly common'?  In general it's a very
niche thing.  But in certain niches it gets used more and more.

> Because page struct
> lookups will become more expensive because we'll have to do type checking and
> Willy may eventually move them from a fixed array into a maple tree - so if we
> can record the P2P flag in the bio_vec, it would help speed up the "not P2P"
> case.

As said before, the best place for that is a higher level structure than
the bio_vec.

> Do we actually need 32 bits for bv_len, especially given that MAX_RW_COUNT is
> capped at a bit less than 2GiB?  Could we, say, do:
> 
>  	struct bio_vec {
>  		phys_addr_t	bv_phys;
>  		u32		bv_len:31;
> 		u32		bv_use_p2p:1;
>  	} __packed;

I've already heard people complain 32-bit might not be enough :) 

> And rather than storing the how-to-do-P2P info in the page struct, does it
> make sense to hold it separately, keyed on bv_phys?

Maybe.  But then you need to invent your own new refcounting for the
section representing the hot pluggable p2p memory.

> Also, is it possible for the networking stack, say, to trivially map the P2P
> memory in order to checksum it?  I presume bv_phys in that case would point to
> a mapping of device memory?

P2P is always to MMIO regions.  So you can access it using the usual
MMIO helpers.