IOMMU Userspace API — The Linux Kernel documentation (2024)

¶

When IOMMU UAPI gets extended, the data structures can only bemodified in two ways:

1. Adding new fields by re-purposing the padding[] field. No size change.

2. Adding new union members at the end. May increase the structure sizes.

No new fields can be added after the variable sized union in that itwill break backward compatibility when offset moves. A new flag mustbe introduced whenever a change affects the structure using eithermethod. The IOMMU driver processes the data based on flags whichensures backward compatibility.

Version field is only reserved for the unlikely event of UAPI upgradeat its entirety.

It’s always the caller’s responsibility to indicate the size of thestructure passed by setting argsz appropriately.Though at the same time, argsz is user provided data which is nottrusted. The argsz field allows the user app to indicate how much datait is providing; it’s still the kernel’s responsibility to validatewhether it’s correct and sufficient for the requested operation.

Compatibility Checking¶

When IOMMU UAPI extension results in some structure size increase,IOMMU UAPI code shall handle the following cases:

1. User and kernel has exact size match

2. An older user with older kernel header (smaller UAPI size) running on anewer kernel (larger UAPI size)

3. A newer user with newer kernel header (larger UAPI size) runningon an older kernel.

4. A malicious/misbehaving user passing illegal/invalid size but withinrange. The data may contain garbage.

Feature Checking¶

While launching a guest with vIOMMU, it is strongly advised to checkthe compatibility upfront, as some subsequent errors happening duringvIOMMU operation, such as cache invalidation failures cannot be nicelyescalated to the guest due to IOMMU specifications. This can lead tocatastrophic failures for the users.

User applications such as QEMU are expected to import kernel UAPIheaders. Backward compatibility is supported per feature flags.For example, an older QEMU (with older kernel header) can run on newerkernel. Newer QEMU (with new kernel header) may refuse to initializeon an older kernel if new feature flags are not supported by olderkernel. Simply recompiling existing code with newer kernel header shouldnot be an issue in that only existing flags are used.

IOMMU vendor driver should report the below features to IOMMU UAPIconsumers (e.g. via VFIO).

1. IOMMU_NESTING_FEAT_SYSWIDE_PASID

2. IOMMU_NESTING_FEAT_BIND_PGTBL

3. IOMMU_NESTING_FEAT_BIND_PASID_TABLE

4. IOMMU_NESTING_FEAT_CACHE_INVLD

5. IOMMU_NESTING_FEAT_PAGE_REQUEST

Take VFIO as example, upon request from VFIO userspace (e.g. QEMU),VFIO kernel code shall query IOMMU vendor driver for the support ofthe above features. Query result can then be reported back to theuserspace caller. Details can be found inVFIO - “Virtual Function I/O” .

Data Passing Example with VFIO¶

As the ubiquitous userspace driver framework, VFIO is already IOMMUaware and shares many key concepts such as device model, group, andprotection domain. Other user driver frameworks can also be extendedto support IOMMU UAPI but it is outside the scope of this document.

In this tight-knit VFIO-IOMMU interface, the ultimate consumer of theIOMMU UAPI data is the host IOMMU driver. VFIO facilitates user-kerneltransport, capability checking, security, and life cycle management ofprocess address space ID (PASID).

VFIO layer conveys the data structures down to the IOMMU driver. Itfollows the pattern below:

struct { __u32 argsz; __u32 flags; __u8 data[];};

Here data[] contains the IOMMU UAPI data structures. VFIO has thefreedom to bundle the data as well as parse data size based on its own flags.

In order to determine the size and feature set of the user data, argszand flags (or the equivalent) are also embedded in the IOMMU UAPI datastructures.

A “__u32 argsz” field is always at the beginning of each structure.

For example:

struct iommu_cache_invalidate_info { __u32 argsz; #define IOMMU_CACHE_INVALIDATE_INFO_VERSION_1 1 __u32 version; /* IOMMU paging structure cache */ #define IOMMU_CACHE_INV_TYPE_IOTLB (1 << 0) /* IOMMU IOTLB */ #define IOMMU_CACHE_INV_TYPE_DEV_IOTLB (1 << 1) /* Device IOTLB */ #define IOMMU_CACHE_INV_TYPE_PASID (1 << 2) /* PASID cache */ #define IOMMU_CACHE_INV_TYPE_NR (3) __u8 cache; __u8 granularity; __u8 padding[6]; union { struct iommu_inv_pasid_info pasid_info; struct iommu_inv_addr_info addr_info; } granu;};

VFIO is responsible for checking its own argsz and flags. It theninvokes appropriate IOMMU UAPI functions. The user pointers are passedto the IOMMU layer for further processing. The responsibilities aredivided as follows:

• Generic IOMMU layer checks argsz range based on UAPI data in thecurrent kernel version.

• Generic IOMMU layer checks content of the UAPI data for non-zeroreserved bits in flags, padding fields, and unsupported version.This is to ensure not breaking userspace in the future when thesefields or flags are used.

• Vendor IOMMU driver checks argsz based on vendor flags. UAPI datais consumed based on flags. Vendor driver has access tounadulterated argsz value in case of vendor specific futureextensions. Currently, it does not perform the copy_from_user()itself. A __user pointer can be provided in some future scenarioswhere there’s vendor data outside of the structure definition.

IOMMU code treats UAPI data in two categories:

• structure contains vendor data(Example: iommu_uapi_cache_invalidate())

• structure contains only generic data(Example: iommu_uapi_sva_bind_gpasid())

Sharing UAPI with in-kernel users¶

For UAPIs that are shared with in-kernel users, a wrapper function isprovided to distinguish the callers. For example,

Userspace caller

int iommu_uapi_sva_unbind_gpasid(struct iommu_domain *domain, struct device *dev, void __user *udata)

In-kernel caller

int iommu_sva_unbind_gpasid(struct iommu_domain *domain, struct device *dev, ioasid_t ioasid);