Re: [PATCH v1 00/23] s390/vfio-ap: Implement live guest migration of guests using AP devices

[Anthony Krowiak]

On 3/30/26 12:27 PM, Alex Williamson wrote:
> On Wed, 25 Mar 2026 17:00:47 -0400
> Anthony Krowiak <akrowiak@xxxxxxxxxxxxx> wrote:
>
> > This patch series implements live guest migration of a guest to which AP
> > devices have been passed through. To better comprehend this design, one has
> > to understand that VFIO AP mediated device is not used to provide userspace
> > with direct access to a device as is the case with other devices that use
> > the VFIO framework to pass them through to a guest. The sole purpose of the
> > VFIO AP mediated device is to manage an AP configuration for a guest. An AP
> > configuration is comprised of the AP adapter IDs (APID), AP queue
> > indexes (APQI) and domain numbers of the control domains to which a guest
> > will be granted access. Once the VFIO AP mediated device is attached to a
> > guest, its AP configuration is set by the vfio_ap device driver. Once set,
> > all access to the AP devices is handled by the s390 Interpretive Execution
> > facility; in other words, the vfio_ap device driver plays no role in
> > providing direct access to the AP devices in the guest's AP configuration.
> >
> > The only role that the vfio_ap device driver plays in the migration
> > process is to verify that the AP configuration for the source guest is
> > compatible with the AP configuration of the destination guest.
> > Incompatibility will result in a live guest migration failure.
> > In order to be compatible, the following requirements must be met:
> >
> > 1. The destination guest will be started with the same QEMU command line
> >     as the source guest, so the mediated device supplying the AP
> >     configuration on both guests must have the same name (UUID).
> AFAIK, same UUID is not a requirement for out-of-tree mdev drivers
> supporting migration.  You're really concerned more with the
> configuration/composition of the mdev device, so requiring the same
> UUID seems a bit arbitrary.

As stated above, the destination guest will be started with the same
QEMU command line as the source guest. Within that command line
will be a '-device' parameter like the following:

-device 
'{"driver":"vfio-ap","id":"hostdev0","sysfsdev":"/sys/bus/mdev/devices/62177883-f1bb-47f0-914d-32a22e3a8804"}

Note that sysfsdev is the path to the mdev named 
62177883-f1bb-47f0-914d-32a22e3a8804;
therefore, the mdev with that name must exist on the destination guest or
the migration will fail with the following error:

error: device not found: mediated device 
'62177883-f1bb-47f0-914d-32a22e3a8804' not found

> > 2. The AP configuration assigned via the VFIO AP mediated device on both
> >     guests must be compatible. As such, each AP configuration must meet
> >     the following requirements:
> >
> >     * Both guests must have the same number of APQNs
> >
> >     * Each APQN assigned to the source guest must also be assigned to the
> >       destination guest
> >
> >     * Each APQN assigned to both guests must reference an AP queue with the
> >       same hardware capabilities
> Why isn't this sufficient vs also requiring the same UUID?
>        
> > Note: There is a forthcoming consumer of this series which will be a QEMU
> >        patch series is entitled:
> >        'hw/vfio/ap: Implement live guest migration of guests using AP
> >        devices'
> >
> > This design also adds a use case for enabling and disabling
> > migration of guests to which AP devices have been passed through. To
> > facilitate this, a new read/write sysfs 'migratable' attribute is added to
> > the mediated device. This attribute specifies whether the vfio device is
> > migratable (1) or not (0). When the value of this attribute is changed, the
> > vfio_ap device driver will signal an eventfd to userspace. It is up to
> > userspace to respond to the change by enabling or disabling migration of
> > the guest to which the mediated device is attached. The operation will be
> > rejected with a 'Device or resource busy' message if a migration is in
> > progress.
> This seems inherently racy.  What happens if the device becomes
> unmigratable while it's being migrated?
>
> If userspace is deciding that the device is now unmigratable, why does
> it need to signal this through the kernel driver rather than with the
> userspace orchestration agent?  The entire path seems unnecessary.
>
> > Userspace must also have a means for retrieving the value of the sysfs
> > 'migratable' attribute when the guest is started to initialize whether it
> > can be migrated. For this, The VFIO_DEVICE_GET_INFO ioctl is used. The
> > struct vfio_device_info object passed to the ioctl will be extended with a
> > capability specifying the vfio device attributes. One of the attributes
> > will contain the value of the mediated device's 'migratable' attribute.
> This is just broken, it's redundant to our current device feature
> mechanism for exposing migration support.  If you want the capability
> to create unmigratable devices statically, can't that be encompassed
> within the definition of the mdev type?  Dynamic migration support just
> seems like it's involving the kernel in orchestration it shouldn't be a
> part of.  Thanks,
>
> Alex
>   
> > Anthony Krowiak (23):
> >    s390/vfio-ap: Store queue hardware info when probed
> >    s390/vfio-ap: Provide access to queue objects and related info
> >    s390/vfio-ap: Add header file for xfer of vfio device caps to
> >      userspace
> >    MAINTAINERS: Add new header file for the S390 VFIO AP DRIVER
> >      maintainers
> >    s390/vfio-ap: A sysfs 'migratable' attribute to enable/disable
> >      migration of guest
> >    s390/vfio-ap: Add 'migratable' feature to sysfs 'features' attribute
> >    s390/vfio-ap: Signal event to enable/disable live guest migration
> >    s390/vfio-ap: Return value of sysfs migratable attribute from
> >      VFIO_DEVICE_GET_INFO ioctl
> >    s390/vfio-ap: Data structures for facilitating vfio device migration
> >    s390/vfio-ap: Initialize/release vfio device migration data
> >    s390-vfio-ap: Callback to set vfio device mig state during guest
> >      migration
> >    s390/vfio-ap: Transition guest migration state from STOP to STOP_COPY
> >    s390/vfio-ap: File ops called to save the vfio device migration state
> >    s390/vfio-ap: Transition device migration state from STOP to RESUMING
> >    s390/vfio-ap: File ops called to resume the vfio device migration
> >    s390/vfio-ap: Transition device migration state from RESUMING to STOP
> >    s390/vfio-ap: Transition device migration state from STOP_COPY to STOP
> >    s390/vfio-ap: Transition device migration state from STOP to RUNNING
> >      and vice versa
> >    s390-vfio-ap: Callback to get the current vfio device migration state
> >    s390/vfio-ap: Callback to get the size of data to be migrated during
> >      guest migration
> >    s390/vfio-ap: Provide API to query whether migration is in progress
> >    s390/vfio-ap: Disallow blocking migration in progress
> >    s390/vfio-ap: Add live guest migration chapter to vfio-ap.rst
> >
> >   Documentation/arch/s390/vfio-ap.rst     |  339 +++++--
> >   MAINTAINERS                             |    1 +
> >   drivers/s390/crypto/Makefile            |    2 +-
> >   drivers/s390/crypto/vfio_ap_drv.c       |    4 +-
> >   drivers/s390/crypto/vfio_ap_migration.c | 1131 +++++++++++++++++++++++
> >   drivers/s390/crypto/vfio_ap_ops.c       |  263 +++++-
> >   drivers/s390/crypto/vfio_ap_private.h   |   20 +
> >   include/uapi/linux/vfio.h               |    2 +
> >   include/uapi/linux/vfio_ap.h            |   34 +
> >   9 files changed, 1685 insertions(+), 111 deletions(-)
> >   create mode 100644 drivers/s390/crypto/vfio_ap_migration.c
> >   create mode 100644 include/uapi/linux/vfio_ap.h