[PATCH v1 24/24] s390/vfio-ap: Add live guest migration chapter to vfio-ap.rst
From: Anthony Krowiak
Date: Wed Mar 25 2026 - 17:13:56 EST
Add a 'Live Guest Migration' chapter to the
Documentation/arch/s390/vfio-ap.rst document to describe the details for
initiating live guest migration for a guest to which AP adapters, domains
and control domains have been passed through.
Signed-off-by: Anthony Krowiak <akrowiak@xxxxxxxxxxxxx>
---
Documentation/arch/s390/vfio-ap.rst | 339 ++++++++++++++++++++--------
1 file changed, 246 insertions(+), 93 deletions(-)
diff --git a/Documentation/arch/s390/vfio-ap.rst b/Documentation/arch/s390/vfio-ap.rst
index eba1991fbdba..9df7614fd62b 100644
--- a/Documentation/arch/s390/vfio-ap.rst
+++ b/Documentation/arch/s390/vfio-ap.rst
@@ -1016,7 +1016,7 @@ guest_matrix dyn ap_config
the following features are advertised:
----------------+---------------------------------------------------------------+
++--------------+---------------------------------------------------------------+
| Flag | Description |
+==============+===============================================================+
| guest_matrix | guest_matrix attribute exists. It reports the matrix of |
@@ -1025,105 +1025,258 @@ the following features are advertised:
+--------------+---------------------------------------------------------------+
| dyn | Indicates hot plug/unplug of AP adapters, domains and control |
| | domains for a guest to which the mdev is attached. |
-+------------+-----------------------------------------------------------------+
++--------------+---------------------------------------------------------------+
| ap_config | ap_config interface for one-shot modifications to mdev config |
+--------------+---------------------------------------------------------------+
+| migratable | Indicates that live guest migration is supported for guests |
+| | to which crypto devices are passed through |
++--------------+---------------------------------------------------------------+
-Limitations
-===========
-Live guest migration is not supported for guests using AP devices without
-intervention by a system administrator. Before a KVM guest can be migrated,
-the vfio_ap mediated device must be removed. Unfortunately, it can not be
-removed manually (i.e., echo 1 > /sys/devices/vfio_ap/matrix/$UUID/remove) while
-the mdev is in use by a KVM guest. If the guest is being emulated by QEMU,
-its mdev can be hot unplugged from the guest in one of two ways:
-
-1. If the KVM guest was started with libvirt, you can hot unplug the mdev via
- the following commands:
-
- virsh detach-device <guestname> <path-to-device-xml>
-
- For example, to hot unplug mdev 62177883-f1bb-47f0-914d-32a22e3a8804 from
- the guest named 'my-guest':
-
- virsh detach-device my-guest ~/config/my-guest-hostdev.xml
-
- The contents of my-guest-hostdev.xml:
-
-.. code-block:: xml
-
- <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-ap'>
- <source>
- <address uuid='62177883-f1bb-47f0-914d-32a22e3a8804'/>
- </source>
- </hostdev>
-
-
- virsh qemu-monitor-command <guest-name> --hmp "device-del <device-id>"
-
- For example, to hot unplug the vfio_ap mediated device identified on the
- qemu command line with 'id=hostdev0' from the guest named 'my-guest':
-
-.. code-block:: sh
-
- virsh qemu-monitor-command my-guest --hmp "device_del hostdev0"
-
-2. A vfio_ap mediated device can be hot unplugged by attaching the qemu monitor
- to the guest and using the following qemu monitor command:
-
- (QEMU) device-del id=<device-id>
-
- For example, to hot unplug the vfio_ap mediated device that was specified
- on the qemu command line with 'id=hostdev0' when the guest was started:
-
- (QEMU) device-del id=hostdev0
-
-After live migration of the KVM guest completes, an AP configuration can be
-restored to the KVM guest by hot plugging a vfio_ap mediated device on the target
-system into the guest in one of two ways:
-
-1. If the KVM guest was started with libvirt, you can hot plug a matrix mediated
- device into the guest via the following virsh commands:
-
- virsh attach-device <guestname> <path-to-device-xml>
-
- For example, to hot plug mdev 62177883-f1bb-47f0-914d-32a22e3a8804 into
- the guest named 'my-guest':
-
- virsh attach-device my-guest ~/config/my-guest-hostdev.xml
-
- The contents of my-guest-hostdev.xml:
-
-.. code-block:: xml
-
- <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-ap'>
- <source>
- <address uuid='62177883-f1bb-47f0-914d-32a22e3a8804'/>
- </source>
- </hostdev>
-
+Live Guest Migration
+====================
+A guest to which crypto devices (i.e., AP adapters, domains and control domains)
+have been passed through can be migrated to another system if both the source
+and destination guests meet the following requirements:
- virsh qemu-monitor-command <guest-name> --hmp \
- "device_add vfio-ap,sysfsdev=<path-to-mdev>,id=<device-id>"
+* The destination guest will be started with the same QEMU command line as the
+ source guest, so the mediated device supplying the AP configuration on both
+ the source and destination guests must have the same name (UUID).
- For example, to hot plug the vfio_ap mediated device
- 62177883-f1bb-47f0-914d-32a22e3a8804 into the guest named 'my-guest' with
- device-id hostdev0:
+* The AP configuration passed through to both the source and destination guests
+ must be compatible. As such, each AP configuration:
- virsh qemu-monitor-command my-guest --hmp \
- "device_add vfio-ap,\
- sysfsdev=/sys/devices/vfio_ap/matrix/62177883-f1bb-47f0-914d-32a22e3a8804,\
- id=hostdev0"
+ * Must have the same number of APQNs
-2. A vfio_ap mediated device can be hot plugged by attaching the qemu monitor
- to the guest and using the following qemu monitor command:
+ * Each APQN assigned to the source guest must also be assigned to the
+ destination guest
- (qemu) device_add "vfio-ap,sysfsdev=<path-to-mdev>,id=<device-id>"
+ * Each APQN assigned to the source guest and destination guest must reference
+ a queue with compatible hardware capabilities:
- For example, to plug the vfio_ap mediated device
- 62177883-f1bb-47f0-914d-32a22e3a8804 into the guest with the device-id
- hostdev0:
++--------------+---------------------------------------------------------------+
+| Hardware | Description |
+| Capabilities | |
++==============+===============================================================+
+| facilities | * AP special command facility (APSC) |
+| | * AP 4096-bit ME PKU commands facility (AP4KM) |
+| | * AP 4096-bit CRT PKU commands (AP4KC) |
++--------------+---------------------------------------------------------------+
+| mode | * CCA-mode |
+| | * Accelerator-mode |
+| | * XCP-mode (EP11) |
++--------------+---------------------------------------------------------------+
+| AP extended | APXA installed |
+| addressing | |
++--------------+---------------------------------------------------------------+
+| command | Command filtering available |
+| filtering | |
++--------------+---------------------------------------------------------------+
+| functional | * Full native card function |
+| capabilities | * Only stateless functions |
++--------------+---------------------------------------------------------------+
+| secure | The guest running on the source host can not have any queues |
+| execution | bound or associated with it |
++--------------+---------------------------------------------------------------+
+| AP type | * No AP type |
+| | * PCICC (Leeds-2) |
+| | * PCICA (Leeds-2 Lite) |
+| | * PCIXCC |
+| | * CEX2A |
+| | * CEX2C |
+| | * CEX3A |
+| | * CEX3C |
+| | * CEX4S |
+| | * CEX5S |
+| | * CEX6S |
+| | * CEX7S |
+| | * CEX8S |
+| | |
+| | Note: The AP type on the source and destination guests can |
+| | differ if the queue passed through to the target guest |
+| | is a newer model (backwards compatible) |
++--------------+---------------------------------------------------------------+
- (QEMU) device-add "vfio-ap,\
- sysfsdev=/sys/devices/vfio_ap/matrix/62177883-f1bb-47f0-914d-32a22e3a8804,\
- id=hostdev0"
+Enabling and disabling live guest migration
+-------------------------------------------
+By default, live guest migration will be disabled for guests that use crypto
+devices. A r/w sysfs 'migratable' attribute for each mediated device is provided
+to enable/disable live guest migration for the guest to which the mediated
+device is attached. The valid values for this attribute are 1 (enable) and
+0 (disable).
+
+To enable live migration:
+
+ echo 1 /sys/bus/mdev/devices/$UUID/migratable
+
+To disable live migration:
+
+ echo 0 /sys/bus/mdev/devices/$UUID/migratable
+
+The 'migratable' attribute allows migration to be dynamically enabled or
+disabled for the guest to which the mediated device is attached; however,
+if migration of the vfio-ap device is already in progress, the operation will
+be rejected with a 'Device or resource busy' message.
+
+Live guest migration failures due to AP configuration errors
+------------------------------------------------------------
+The VFIO AP mediated device is not used to provide userspace with direct access
+to a device as is the case with other devices that use the VFIO framework to
+pass them through to a guest. The sole purpose of the VFIO AP mediated device
+is to manage an AP configuration for a guest. An AP configuration is comprised
+of the AP adapter IDs (APID), AP queue indexes (APQI) and domain numbers of the
+control domains to which a guest will be granted access. Once the VFIO AP
+mediated device is attached to the guest, its AP configuration is set by the
+vfio_ap device driver. Once set, all access to the AP devices is handled
+by the s390 Interpretive Execution facility; in other words, the vfio_ap device
+driver plays no role in providing direct access to these AP devices.
+
+The only role that the vfio_ap device driver plays in the migration process is
+to verify that the AP configuration for the source guest is compatible with the
+AP configuration of the destination guest. Incompatibility will result in a live
+guest migration failure. This section describes how to debug this type of
+failure.
+
+The destination host is missing the mediated device with the same name as the
+mdev attached to the source guest
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+**Source host**
+
++--------------+--------------------------------------------------------------------+
+| Log | Message |
++==============+====================================================================+
+| Console log: | *error: device not found: mediated device '$UUID' not found* |
++--------------+--------------------------------------------------------------------+
+
+The source guest is not enabled for migration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+**Source host**
+
++--------------+--------------------------------------------------------------------+
+| Log | Message |
++==============+====================================================================+
+| Console log: | *error: Requested operation is not valid: |
+| | cannot migrate domain: $UUID: Migration is disabled for VFIO |
+| | device* |
++--------------+--------------------------------------------------------------------+
+
+
+The AP configuration of the source and destination guests are not compatible
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+**Source host**
+
++--------------+--------------------------------------------------------------------+
+| Log | Message |
++==============+====================================================================+
+| Console log: | *error: operation failed: job 'migration in' failed: load of |
+| | migration failed: Bad address* |
++--------------+--------------------------------------------------------------------+
+| Kernel log: | N/A |
++--------------+--------------------------------------------------------------------+
+| QEMU log: | *initiating migration* |
+| | |
+| | *qemu-system-s390x: Sibling indicated error 1* |
++--------------+--------------------------------------------------------------------+
+
+**Destination host**
+
++--------------+--------------------------------------------------------------------+
+| Log | Message |
++==============+====================================================================+
+| Console log: | N/A |
++--------------+--------------------------------------------------------------------+
+| Kernel log: | *vfio_ap_mdev $UUID: Migration failed: Source and target queues |
+| | ($APQN) are not compatible* |
+| | |
+| | The message above will be followed by one or more messages |
+| | enumerating the incompatible features; for example: |
+| | |
+| | *vfio_ap_mdev $UUID: APSC facility installed in source queue $APQN*|
+| | |
+| | *vfio_ap_mdev $UUID: APSC facility not installed in target queue |
+| | $APQN* |
+| | |
+| | *AP type of source ($APTYPE) not compatible with target ($APTYPE)* |
++--------------+--------------------------------------------------------------------+
+| QEMU log: | *initiating migration* |
+| | |
+| | *qemu-system-s390x: error while loading state section id ...* |
+| | |
+| | *shutting down, reason=failed* |
+| | |
+| | *terminating on signal 15 from pid 1196 (/usr/sbin/virtqemud)* |
++--------------+--------------------------------------------------------------------+
+
+The AP configuration of the source guest has more APQNS than the destination guest
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+**Source host**
+
++--------------+--------------------------------------------------------------------+
+| Log | Message |
++==============+====================================================================+
+| Console log: | *error: operation f1e166ee77e6failed: job 'migration in' failed: load of |
+| | migration failed: Input/output error* |
++--------------+--------------------------------------------------------------------+
+| Kernel log: | N/A |
++--------------+--------------------------------------------------------------------+
+| QEMU log: | *initiating migration* |
+| | |
+| | *qemu-system-s390x: Sibling indicated error 1* |
++--------------+--------------------------------------------------------------------+
+
+**Destination host**
+
++--------------+--------------------------------------------------------------------+
+| Log | Message |
++==============+====================================================================+
+| Console log: | N/A |
++--------------+--------------------------------------------------------------------+
+| Kernel log: | *vfio_ap_mdev $UUID: |
+| | migration failed: source guest's AP config size (xx) larger than |
+| | target's (yy)* |
++--------------+--------------------------------------------------------------------+
+| QEMU log: | *initiating migration* |
+| | |
+| | *qemu-system-s390x: error while loading state section id ...* |
+| | |
+| | *shutting down, reason=failed* |
+| | |
+| | *terminating on signal 15 from pid 1196 (/usr/sbin/virtqemud)* |
++--------------+--------------------------------------------------------------------+
+
+The AP configuration of the source guest has fewer APQNS than the destination guest
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+**Source host**
+
++--------------+--------------------------------------------------------------------+
+| Log | Message |
++==============+====================================================================+
+| Console log: | *error: operation failed: job 'migration in' failed: load of |
+| | migration failed: No such device* |
++--------------+--------------------------------------------------------------------+
+| Kernel log: | N/A |
++--------------+--------------------------------------------------------------------+
+| QEMU log: | *initiating migration* |
+| | |
+| | *qemu-system-s390x: Sibling indicated error 1* |
++--------------+--------------------------------------------------------------------+
+
+**Destination host**
+
++--------------+--------------------------------------------------------------------+
+| Log | Message |
++==============+====================================================================+
+| Console log: | N/A |
++--------------+--------------------------------------------------------------------+
+| Kernel log: | *vfio_ap_mdev $UUID: |
+| | migration failed: number of queues on source (x) and target (y) |
+| | guests differ* |
++--------------+--------------------------------------------------------------------+
+| QEMU log: | *initiating migration* |
+| | |
+| | *qemu-system-s390x: error while loading state section id ...* |
+| | |
+| | *shutting down, reason=failed* |
+| | |
+| | *terminating on signal 15 from pid 1196 (/usr/sbin/virtqemud)* |
++--------------+--------------------------------------------------------------------+
--
2.52.0