[PATCH 00/14] media: apple: add avd driver

[Sofus Forstreuter]

Hi,

This series enables hardware video decoding on Apple Silicon M1, M2 and
M3 SoCs. It adds support for all the codecs AVD (Apple Video Decoder)
supports, which is H264, H265, VP9 and AV1 (only on M3+).

The AVD is a bit unique both as a video decoder and an IP block on Apple
Silicon SoCs. Instead of following the normal "a register for each
parameter", AVD is programmed through one single register. This means
the order of the writes are important, but its also important to ensure
we dont write faster than the hardware can process.
To solve this problem the driver writes the 'instructions' into a list
of 'segments'. A shared function then submits those to the hardware
while making sure to wait each time the hardware lacks behind.

The block also includes Cortex-M3 co-processor that accepts unsigned
code. The CM3 is not really required, but is the only way to get decode
status IRQ's. For now, the CM3 only handles decode-tunable configuration
and forwards all IRQ's to the AP [1]. In the future this will be
extended to handle the instruction submission, this would likely result
in decreased power consumption, some initial testing even suggests
slightly faster decode times.

The driver implements the V4L2 M2M stateless decoder API. The driver's
framework is based very heavy on rkvdecs driver. It therefore comes as
no suprise that `v4l2-compliance` reports no errors:
Total for avd device /dev/video0: 49, Succeeded: 49, Failed: 0, Warnings: 0

The fluster scores are:
JVT-AVC_V1: 77/135
JVT-FR-EXT: 42/69
VP9-TEST-VECTORS: 216/305
VP9-TEST-VECTORS-HIGH: 1/6
JCT-VC-HEVC_V1: 143/147
AV1-TEST-VECTORS: 238/242

Since not all unsupported H264 streams are rejected, the scores
depend on the reset working, this only works with a patch to the
apple-dart driver so that the IOMMU attach/detach trick works.

All codecs support 4:2:2, 4:2:0, in 10 and 8 bit. AVD outputs in NV12,
NV16, P010 and P210. Support for 4:4:4 and 12 bit formats is not added
yet.

Internally the AVD uses an Apple specific compressed and tiled format
called Interchange to store reference image data. This is stored after
image data across all codecs. Apple Interchange is used for sharing
buffers with other hardware blocks (GPU, display controller), support
for this is on its way [2].

This driver has been tested on most Apple Silicon SoCs in our downstream
tree [3]. Its expected this driver will work on M4, M5, M6 and the Neo,
with firmware already being written for all except the M6 SoCs [1].

The groundwork for this was laid many years ago by Jamie, R and Eileen,
who through a combined effort managed to reverse engineer many core
parts of the AVD block.
This driver would not have been possible without the work done by
Eileen, who wrote initial support for H264, H265, VP9 and excellent
tooling [4], allowing me to reverse engineer AV1 and make the driver
conformant.

[1] https://github.com/AsahiLinux/avd-fw
[2] https://patch.msgid.link/20260917-apple-interchange-modifier-v1-1-875294689c48@xxxxxxxxx
[3] https://github.com/AsahiLinux/linux
[4] https://github.com/eiln/avd/tree/main

Signed-off-by: Sofus Forstreuter <sofus.c@xxxxxxxxxx>
---
Sofus Forstreuter (14):
      media: v4l2: Add P210 pixel format
      dt-bindings: media: add apple,avd
      arm64: dts: apple: t8103: add avd nodes
      arm64: dts: apple: t8112: add avd nodes
      arm64: dts: apple: t8122: add avd nodes
      arm64: dts: apple: t600x: add avd nodes
      arm64: dts: apple: t602x: add avd nodes
      arm64: dts: apple: t6030: add avd nodes
      arm64: dts: apple: t6031: add avd nodes
      media: apple: add avd driver
      media: apple: avd: add h264 support
      media: apple: avd: add vp9 support
      media: apple: avd: add hevc support
      media: apple: avd: add av1 support

 .../devicetree/bindings/media/apple,avd.yaml       |  112 +
 .../userspace-api/media/v4l/pixfmt-yuv-planar.rst  |   10 +-
 MAINTAINERS                                        |    2 +
 arch/arm64/boot/dts/apple/t600x-dieX.dtsi          |   24 +
 arch/arm64/boot/dts/apple/t602x-dieX.dtsi          |   24 +
 arch/arm64/boot/dts/apple/t6030.dtsi               |   24 +
 arch/arm64/boot/dts/apple/t6031-dieX.dtsi          |   24 +
 arch/arm64/boot/dts/apple/t8103.dtsi               |   24 +
 arch/arm64/boot/dts/apple/t8112.dtsi               |   24 +
 arch/arm64/boot/dts/apple/t8122.dtsi               |   24 +
 drivers/media/platform/Kconfig                     |    1 +
 drivers/media/platform/Makefile                    |    1 +
 drivers/media/platform/apple/Kconfig               |    5 +
 drivers/media/platform/apple/Makefile              |    3 +
 drivers/media/platform/apple/avd/Kconfig           |   18 +
 drivers/media/platform/apple/avd/Makefile          |    5 +
 .../media/platform/apple/avd/avd-av1-entropymode.c | 4698 ++++++++++++++++++++
 .../media/platform/apple/avd/avd-av1-entropymode.h |  284 ++
 drivers/media/platform/apple/avd/avd-av1.c         | 1447 ++++++
 drivers/media/platform/apple/avd/avd-drv.c         |  817 ++++
 drivers/media/platform/apple/avd/avd-h264.c        |  851 ++++
 drivers/media/platform/apple/avd/avd-hevc.c        | 1326 ++++++
 drivers/media/platform/apple/avd/avd-hw.c          |  127 +
 drivers/media/platform/apple/avd/avd-inst.h        |  218 +
 drivers/media/platform/apple/avd/avd-v4l2.c        |  962 ++++
 drivers/media/platform/apple/avd/avd-vp9.c         | 1072 +++++
 drivers/media/platform/apple/avd/avd.h             |  305 ++
 drivers/media/v4l2-core/v4l2-common.c              |    1 +
 drivers/media/v4l2-core/v4l2-ioctl.c               |    1 +
 include/uapi/linux/videodev2.h                     |    1 +
 30 files changed, 12434 insertions(+), 1 deletion(-)
---
base-commit: aedd77ea8168f00687c1248cdf115071e869cf93
change-id: 20260917-avd-7ee5474b468c

Best regards,
--  
Sofus Forstreuter <sofus.c@xxxxxxxxxx>