RE: [RFC PATCH v4 03/16] iommu/arm-smmu-v3: Add initial pSMMU realm viommu plumbing

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@xxxxxxxx>
> Sent: Tuesday, September 15, 2026 9:43 PM
> 
> On Tue, Sep 15, 2026 at 07:13:48AM +0000, Tian, Kevin wrote:
> > > Yes, this already has to be true or our lifecylce model is
> > > nonsense. Once the iommufd creates the vdevice the guest is running
> > > and we cannot disconnect it without unplugging it from the guest. Thus
> > > the locking can rely on that. Within an iommufd ioctl context the vdev
> > > and underlying connection must be stable.
> >
> > btw looks a related suggestion is having the tsm driver (arm-cca-host
> > here) implement the viommu ops while leaving the original smmu driver
> > largely intact, and "obtain the viommu through tsm_ops not through
> > iommu_ops".
> >
> > It's not super clear to me about the implication behind after reading
> > related discussions and what the "obtain" part means.
> >
> > Could you help elaborate this part?
> >
> > The old way is kind of:
> >
> > vdevice_tsm_ops -> tsm_ops -> tsm driver -> iommu driver
> >
> > then it will become:
> >
> > viommu_ops (tsm related) -> tsm_driver
> >
> > where tsm driver will fully handle tsm related viommu_ops?
> 
> At least for ARM there is effectively no entanglement with the actual
> host iommu driver. The viommu is entirely provided by software in the
> RMM world, so it can have its own dedicated driver. In ARM T=1
> transactions are alwayus routed to the RMM's iommu and there is no
> relation to the host.
> 
> I am interested how Intel works here, but I thought it was similar.

Largely yes. Main difference at Intel side is that TDX still relies on the
host to initiate iotlb invalidation (upon notification from KVM on S-EPT
change). Currently we put this logic in intel-iommu driver but it's more
about wrapping invalidation info and passing it to the firmware. Moving
it into the tsm driver should be straightforward.

Maybe there'll be other subtle connections to host iommu driver but
it doesn't sound a hard problem to solve.

> 
> AMD is different and I suspect AMD will have to continue to use the
> viommu from the AMD iommu driver, but I am not sure.

ARM/Intel may support guest viommu in the future. So AMD's case is
a good reference. Maybe for ARM/Intel we'll stay with two or more
viommu ops: one from tsm driver (TDI w/o guest viommu), 2nd from
the host iommu driver (legacy w/ guest viommu), 3rd from the host
iommu driver (TDI w/ guest viommu), etc. We'll see...

> 
> > btw I held the impression of blocked operations from past discussion [1].
> > Initial attempt tried to proactively unbind the TDI upon any operations
> > which may transit the TDI to the ERROR state. Then the suggestion at
> > the moment was:
> 
> With this construction the iommufd vdev is created and permanently
> exist as long as the VM and viommu exist.
> 
> How/when the tsm driver links this to a arch specific "bind/unbind"
> operation is more up to that driver, but I would expect what is
> thought of as "bind" should be the affiliation of the device's T=1
> stream with the viommu and the target VM. It should not be sensitive
> to the TDISP state.

Not sure about this part.

Each arch has its own definition about the binding flow (about 'how'),
but sharing a common step by sending TDISP message to transit the
TDI into the CONFIG_LOCKED state upon guest request (i.e. 'when').

According to the TDISP spec, memory reads/writes with T bit set is
accepted only when the TDI is in RUN state (except MSI/MSI-X writes
are allowed with T bit set in LOCKED but I don't think any arch supports
it yet).

So your definition of 'bind' essentially affiliate it to the RUN state?

> 
> That is not prohibited, the TSM driver could do some auto
> "bind/unbind" whatever that means triggered by ops or tdisp state
> changing under the covers. But this cannot leak out as some kind of
> asynchronous vdev destruction.

Maybe it'd be clearer using an example e.g. ARM to clarify the
suggested split. Or wait for Aneesh's next version...

> 
> > " But now the suggestion is never let VFIO do unbind, instead VFIO
> > should block these operations when device is bound. "
> >
> > [1] https://lore.kernel.org/all/aEFmPaYorqaYCKBY@yilunxu-OptiPlex-7050/
> 
> IDK, if the host wants to put a device into error I don't see why the
> kernel should block it? There are many reasons a device can reach
> error, that needs to be handled.
> 

hmm that's a good point!