Re: [PATCH RFC 04/20] Documentation: ctlv: Add userspace ABI

[Jason Gunthorpe]

> Describe the CTLV transport, discovery, wire format and userspace
> compatibility rules.
> 
> Assisted-by: Cursor: GPT-5.6 Sol
> Assisted-by: Cursor: Claude Opus 5.5
> Signed-off-by: Jiri Pirko <jiri@xxxxxxxxxx>
> ---
>  Documentation/userspace-api/ctlv.rst  | 507 ++++++++++++++++++++++++++
>  Documentation/userspace-api/index.rst |   1 +
>  MAINTAINERS                           |   1 +
>  3 files changed, 509 insertions(+)
>  create mode 100644 Documentation/userspace-api/ctlv.rst
>
> diff --git a/Documentation/userspace-api/ctlv.rst b/Documentation/userspace-api/ctlv.rst
> new file mode 100644
> index 00000000000000..5ebca3d4d94af8
> --- /dev/null
> +++ b/Documentation/userspace-api/ctlv.rst
> @@ -0,0 +1,507 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +==================
> +CTLV userspace ABI
> +==================
> +
> +:Author: Jiri Pirko
> +

This could use an introduction:

Char-Device Tag Length Value (CTLV) is a flexible way to describe system calls
delivered over ioctl for a kernel subsystem. It provides a 64 bit operation
number with variable length input/output arguments described in a YAML schema.
The arguments are serialized using a TLV scheme. Strong support for
forwards/backward compatability is baked into the schema design.

CTLV excels for uAPIs that are too unconstrained to fit in simple structs.
Things that have arrays of arrays, variable size structs, variable sized data,
strings and similar which are challenging to represent.

As a transport CTLV uses a global number space. Inspection tools like
strace/seccomp can always decode a CTLV ioctl without having to know what
subsystem is using it. The global number space is organized into families, which
roughly corresponds to a subsystem.

A family describes what its devices accept in YAML, a generator turns that into
kernel metadata and UAPI headers, and CTLV validates every message before a
family callback sees it. Userspace sees one character device per instance, one
ioctl, and ``read()`` for events.

> +CTLV is a transport for device control interfaces. A family describes what its
> +devices accept in YAML, a generator turns that into kernel metadata and UAPI
> +headers, and CTLV validates every message before a family callback sees it.
> +Userspace sees one character device per instance, one ioctl, and ``read()`` for
> +events.
> +
> +This document is the ABI. What each family accepts is specified under
> +``Documentation/ctlv/specs``; writing a family is :doc:`/driver-api/ctlv`.
> +
> +Why a TLV transport
> +===================
> +
> +CTLV puts the rules of every operation in one description and validates them
> +once:
> +
> +- One ioctl carries every operation of every family, so a new operation is a
> +  new specification entry, not a new ioctl number.
> +- Attributes are optional or required by declaration, so a message can grow
> +  without breaking a caller that does not know the new attribute.
> +- An unknown attribute is refused on a request and tolerated on a reply, so a
> +  new kernel can say more to an old program. A request may mark an attribute an
> +  old kernel is welcome to skip, so a new program can say more too.
> +- Bounds, lengths, occurrence counts and nesting depth are checked before a
> +  family callback runs.
> +
> +Devices
> +=======
> +
> +Each registered instance appears as a character device in the ``ctlv`` class
> +with a dynamic minor::
> +
> +    /dev/ctlv0
> +    /sys/class/ctlv/ctlv0
> +
> +Sysfs shows class membership, the ``dev`` number and the device-model link to
> +the parent device. There is no family sysfs attribute; ``families-dump`` on an
> +open descriptor is the authoritative answer to what a device is.

I've tried to like this but, I really am not a fan. I think it
reflects too much of a netlink influence.

My two biggest problems:

1) This prevents us from incrementally using CTLV as a more robust
   uAPI. Like if I wanted to start using it in vfio for new uAPI I
   cannot

2) It destroys the kernel concept of a cdev subsystem, where we have a
   distinct class, /dev/ directory and cdev nodes.

   Lots of tooling and userspace is geared toward working this way. A
   completely unstable /dev/ctlvX that could be *anything* will need a
   meaningful new methodology in userspace. For instance anyone doing
   containers that wants to pass these is going to need special
   tooling.

> [ ... 143 lines skipped ... ]
> +An attribute ID names the family that declared it, a group within that family
> +and an attribute within that group::
> +
> +    63           48 47           32 31                             0
> +    +--------------+--------------+--------------------------------+
> +    |    family    |    group     |             attr               |
> +    +--------------+--------------+--------------------------------+
> +
> +    CTLV_ATTR(family, group, attr)
> +    CTLV_ATTR_FAMILY(attr) / CTLV_ATTR_GROUP(attr) / CTLV_ATTR_NUMBER(attr)
> +
> +An operation ID is the same shape with the group bits zero::
> +
> +    CTLV_OP(family, op)
> +    CTLV_OP_FAMILY(op) / CTLV_OP_NUMBER(op)
> +
> +Use generated complete IDs. A family's header defines every ID it declares
> +and includes its parent's.

I'm OK with this, it seems like a fine design, very netlink inspired.

One of the nice properties is an external tool, like strace or a
seccomp does have enough information to capture the entire operation,
including all parameters.

Though I wonder how much trouble it will be for userspace to form
these things.. It has to pre-determine how much memory it needs then
fill in this structure. That's not so easy to use.

How hard would it be to accept a linked list, or array, of buffer
frags? With some strict rule that a leaf attribute cannot span frags?

My thinking is userspace could allocate frag by frag as it builds the
message instead of being forced to do a size, allocate, fill sequence?

> +
> +Request occurrence order
> +------------------------
> +
> +Occurrences of one attribute ID must be contiguous within each request root or
> +nest. ``A A B`` is valid, while ``A B A`` is not. An ignored unknown attribute
> +is not an occurrence of any known attribute, but it ends the preceding run, so
> +``A ignored-unknown A`` is also invalid. Different IDs otherwise have no
> +required numeric or schema order.

I'm not entirely sure what this is trying to say. Is it just a single
ID must be contiguous if it is repeated?

> +
> +Attributes a kernel may skip
> +----------------------------
> +
> +A request attribute, the root aside, may carry
> +``CTLV_ATTR_F_IGNORE_IF_UNKNOWN``. If no family the device answers declares
> +that complete ID, the kernel steps over the whole attribute without looking
> +inside it; without the bit, an attribute the schema does not place where it
> +appears fails the request as it always has.
> +
> +The bit forgives an ID this kernel has never heard of and nothing else. An ID
> +the device's schema declares somewhere is held to that schema, so an attribute
> +sent where it does not belong fails whether the bit is set or not, and one sent
> +where it does belong is checked as usual. Every other rule applies too: the
> +length has to fit its nest and the header may carry no bit the format does not
> +define. Replies and events never carry it.

This is pretty hard to read. AI?

> +
> +This is how a program built against a newer version of a family talks to an
> +older kernel. What that kernel knows is validated, what it does not is
> +ignored, and a caller that has to know which is which asks ``op-schema-get``
> +first.
> +
> +Executing an operation
> +======================
> +
> +One ioctl executes every operation::
> +
> +    struct ctlv_op_exec {
> +            __aligned_u64 op_id;
> +            __aligned_u64 request;      /* struct ctlv_attr __user * */
> +            __aligned_u64 reply;        /* struct ctlv_attr __user * */
> +            __u32 request_len;
> +            __u32 reply_len;
> +            __aligned_u64 generation;   /* queries only */
> +            __aligned_u64 cursor;       /* queries only */
> +            __aligned_u64 reserved[2];  /* must be zero */
> +    };
> +
> +    #define CTLV_IOC_OP_EXEC _IOWR(0xBB, 0x00, struct ctlv_op_exec)

Generally I've ben frowning on using the size arugment for ioctl. This
means you can never make that struct bigger, and that's historically
been a bad choice...

I've mostly been using this:

  #define IOMMU_DESTROY _IO(IOMMUFD_TYPE, IOMMUFD_CMD_DESTROY)

And then put a length at the top of the struct and do the

        ret = copy_struct_from_user(ucmd.cmd, op->size, ucmd.ubuffer,
                                    ucmd.user_size);

Kind of thing so there is a solid compatability story.

> [ ... 81 lines skipped ... ]
> +``ENOMEM``
> +    The kernel could not allocate what the operation needed.
> +

EOVERFLOW for check_overflow failures

-- 
Jason