RE: [RFC PATCH v4 03/16] iommu/arm-smmu-v3: Add initial pSMMU realm viommu plumbing

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@xxxxxxxx>
> Sent: Wednesday, September 16, 2026 8:40 PM
> 
> On Wed, Sep 16, 2026 at 05:54:57AM +0000, Tian, Kevin wrote:
> > > At least for ARM there is effectively no entanglement with the actual
> > > host iommu driver. The viommu is entirely provided by software in the
> > > RMM world, so it can have its own dedicated driver. In ARM T=1
> > > transactions are alwayus routed to the RMM's iommu and there is no
> > > relation to the host.
> > >
> > > I am interested how Intel works here, but I thought it was similar.
> >
> > Largely yes. Main difference at Intel side is that TDX still relies on the
> > host to initiate iotlb invalidation (upon notification from KVM on S-EPT
> > change). Currently we put this logic in intel-iommu driver but it's more
> > about wrapping invalidation info and passing it to the firmware. Moving
> > it into the tsm driver should be straightforward.
> >
> > Maybe there'll be other subtle connections to host iommu driver but
> > it doesn't sound a hard problem to solve.
> 
> Okay, so I saw the driver posting for basic iommu support, can we try
> to rework that to be split out like Aneesh is doing so everything
> about TDX calls lives in tsm and intel iommu only provides a small API
> surface to exchange whatever details are needed to bootstrap TDX
> module?

Yeah that makes sense.

> 
> > > AMD is different and I suspect AMD will have to continue to use the
> > > viommu from the AMD iommu driver, but I am not sure.
> >
> > ARM/Intel may support guest viommu in the future.
> 
> ARM supports guest viommu today, it is in the public spec. Secure

sure, I meant the secure part. :)

> guest vSMMU is entirely handled inside the RMM and has no connection
> to the host iommu driver. It is a <100 line ++ on top of Aneesh's
> work, Nicolin posted a draft at one point in those threads.
> 
> I anticipate a future intel guest T=1 viommu should be the same.

I expect so.

> 
> Thus I expect Intel/ARM to have two viommus, one that handles the T=1
> stream owned by the TSM driver and implemented entirely by calling
> TDX/RMM.

and this one is special, not tying to existing IOAS/HWPT/attach/detach/etc.
semantics.

> 
> One that handles the T=0 stream owned by the iommu driver - and it
> already exists.

sounds a clear split.

> 
> > So AMD's case is a good reference.
> 
> I think, AMD is completely different. I keep forgetting thier thing,
> but IIRC they have a secure DTE but instead of having the secure word
> control the translation it controls the RMP and you end up using the
> host's translation for T=1 traffic. This is fundamentally different
> from how Intel and ARM are doing it where the actually IOVA translate
> is under the control of the secure world.

that reminds me...

> 
> Both Intel and ARM put the S-EPT into the iommu HW directly.
> 
> So, I expect Intel to have an API similar to ARM. When you create the
> TSM viommu you tell it if the TDX module should create a secure guest
> visible VT-d emulation. TDX module has to perform the entire emulation
> because it must be trusted. Existing viommu ops should cover the
> remaining to register pdevices as vdevices, provide the vBDF and so
> on.

will think more along that line.

> 
> > > How/when the tsm driver links this to a arch specific "bind/unbind"
> > > operation is more up to that driver, but I would expect what is
> > > thought of as "bind" should be the affiliation of the device's T=1
> > > stream with the viommu and the target VM. It should not be sensitive
> > > to the TDISP state.
> >
> > Not sure about this part.
> >
> > Each arch has its own definition about the binding flow (about 'how'),
> > but sharing a common step by sending TDISP message to transit the
> > TDI into the CONFIG_LOCKED state upon guest request (i.e. 'when').
> 
> Sure, the LOCKED command can be relayed from the guest, but that
> shouldn't be called BIND. locked/unlock/run/err is taking a iommufd
> vdev that is already affiliated with the VM to a specific TDISP state
> 
> > According to the TDISP spec, memory reads/writes with T bit set is
> > accepted only when the TDI is in RUN state (except MSI/MSI-X writes
> > are allowed with T bit set in LOCKED but I don't think any arch supports
> > it yet).
> 
> Sure
> 
> > So your definition of 'bind' essentially affiliate it to the RUN state?
> 
> No, it is informing the secure world that a physical PCI function is
> now a virtual PCI function, is a TDI, and is in a certain VM.
> 
> Outside virtual hotplug this is a permanent action when the VM is
> created.
> 
> > > That is not prohibited, the TSM driver could do some auto
> > > "bind/unbind" whatever that means triggered by ops or tdisp state
> > > changing under the covers. But this cannot leak out as some kind of
> > > asynchronous vdev destruction.
> >
> > Maybe it'd be clearer using an example e.g. ARM to clarify the
> > suggested split. Or wait for Aneesh's next version...
> 
> In ARM:
> 
> BIND is RMI_VDEV_CREATE it links a physical device to a virtual
> device in a realm.

TDX: TDH.TDI.CREATE

> 
> RMI_VSMMU_CREATE can attach a vSMMU to the realm and there is some way
> to link the VDEV And the VSMMU together

TDX: TDH.DMAR.ADD

> 
> Some sequence of RMI_VDEV_COMMUNICATE, RMI_VDEV_LOCK,
> RMI_VDEV_UNLOCK
> and a few others manipulate the UNLOCKED/LOCKED/RUN/ERR TDISP state of
> the VDEV.

to locked: TDH.TDI.BIND
to unlocked: TDH.TDI.UNBIND
to RUN: TDH.TDI.START

plus some other commands required around those points

> 
> I assume TDX has the same general shape, I don't know how you could
> implement this in a radically different way?

yes it's similar.

> 
> So iommufd viommu create calls RMI_VSMMU_CREATE
>    iommufd vdev create calls RMI_VDEV_CREATE
>    iommufd viommu op ioctl calls the COMMUNICATE/LOCK/UNLOCK
> 

so in this case BIND is essentially CREATE. Probably link/affiliate is a
clearer name in description to avoid confusion with vendor specific
bind command...