/ appendices / VK_EXT_image_drm_format_modifier.txt
VK_EXT_image_drm_format_modifier.txt
1 // Copyright (c) 2018-2019 Khronos Group. This work is licensed under a 2 // Creative Commons Attribution 4.0 International License; see 3 // http://creativecommons.org/licenses/by/4.0/ 4 5 include::meta/VK_EXT_image_drm_format_modifier.txt[] 6 7 *Last Modified Date*:: 8 2018-08-29 9 *IP Status*:: 10 No known IP claims. 11 *Contributors*:: 12 - Antoine Labour, Google 13 - Bas Nieuwenhuizen, Google 14 - Chad Versace, Google 15 - James Jones, NVIDIA 16 - Jason Ekstrand, Intel 17 - Jőrg Wagner, ARM 18 - Kristian Høgsberg Kristensen, Google 19 - Ray Smith, ARM 20 21 === Overview 22 23 ==== Summary 24 25 This extension provides the ability to use _DRM format modifiers_ with 26 images, enabling Vulkan to better integrate with the Linux ecosystem of 27 graphics, video, and display APIs. 28 29 Its functionality closely overlaps with 30 `EGL_EXT_image_dma_buf_import_modifiers`^<<VK_EXT_image_drm_format_modifier-fn2,2>>^ 31 and 32 `EGL_MESA_image_dma_buf_export`^<<VK_EXT_image_drm_format_modifier-fn3,3>>^. 33 Unlike the EGL extensions, this extension does not require the use of a 34 specific handle type (such as a dma_buf) for external memory and provides 35 more explicit control of image creation. 36 37 ==== Introduction to DRM Format Modifiers 38 39 A _DRM format modifier_ is a 64-bit, vendor-prefixed, semi-opaque unsigned 40 integer. 41 Most _modifiers_ represent a concrete, vendor-specific tiling format for 42 images. 43 Some exceptions are etext:DRM_FORMAT_MOD_LINEAR (which is not 44 vendor-specific); etext:DRM_FORMAT_MOD_NONE (which is an alias of 45 etext:DRM_FORMAT_MOD_LINEAR due to historical accident); and 46 etext:DRM_FORMAT_MOD_INVALID (which does not represent a tiling format). 47 The _modifier's_ vendor prefix consists of the 8 most significant bits. 48 The canonical list of _modifiers_ and vendor prefixes is found in 49 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/drm/drm_fourcc.h[`drm_fourcc.h`] 50 in the Linux kernel source. 51 The other dominant source of _modifiers_ are vendor kernel trees. 52 53 One goal of _modifiers_ in the Linux ecosystem is to enumerate for each 54 vendor a reasonably sized set of tiling formats that are appropriate for 55 images shared across processes, APIs, and/or devices, where each 56 participating component may possibly be from different vendors. 57 A non-goal is to enumerate all tiling formats supported by all vendors. 58 Some tiling formats used internally by vendors are inappropriate for 59 sharing; no _modifiers_ should be assigned to such tiling formats. 60 61 Modifier values typically do not _describe_ memory layouts. 62 More precisely, a _modifier_'s lower 56 bits usually have no structure. 63 Instead, modifiers _name_ memory layouts; they name a small set of 64 vendor-preferred layouts for image sharing. 65 As a consequence, in each vendor namespace the modifier values are often 66 sequentially allocated starting at 1. 67 68 Each _modifier_ is usually supported by a single vendor and its name matches 69 the pattern `{VENDOR}_FORMAT_MOD_*` or `DRM_FORMAT_MOD_{VENDOR}_*`. 70 Examples are etext:I915_FORMAT_MOD_X_TILED and 71 etext:DRM_FORMAT_MOD_BROADCOM_VC4_T_TILED. 72 An exception is etext:DRM_FORMAT_MOD_LINEAR, which is supported by most 73 vendors. 74 75 Many APIs in Linux use _modifiers_ to negotiate and specify the memory 76 layout of shared images. 77 For example, a Wayland compositor and Wayland client may, by relaying 78 _modifiers_ over the Wayland protocol `zwp_linux_dmabuf_v1`, negotiate a 79 vendor-specific tiling format for a shared code:wl_buffer. 80 The client may allocate the underlying memory for the code:wl_buffer with 81 GBM, providing the chosen _modifier_ to code:gbm_bo_create_with_modifiers. 82 The client may then import the code:wl_buffer into Vulkan for producing 83 image content, providing the resource's dma_buf to 84 slink:VkImportMemoryFdInfoKHR and its _modifier_ to 85 slink:VkImageDrmFormatModifierExplicitCreateInfoEXT. 86 The compositor may then import the code:wl_buffer into OpenGL for sampling, 87 providing the resource's dma_buf and _modifier_ to code:eglCreateImage. 88 The compositor may also bypass OpenGL and submit the code:wl_buffer directly 89 to the kernel's display API, providing the dma_buf and _modifier_ through 90 code:drm_mode_fb_cmd2. 91 92 ==== Format Translation 93 94 _Modifier_-capable APIs often pair _modifiers_ with DRM formats, which are 95 defined in 96 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/drm/drm_fourcc.h[`drm_fourcc.h`]. 97 However, `VK_EXT_image_drm_format_modifier` uses elink:VkFormat instead of 98 DRM formats. 99 The application must convert between elink:VkFormat and DRM format when it 100 sends or receives a DRM format to or from an external API. 101 102 The mapping from elink:VkFormat to DRM format is lossy. 103 Therefore, when receiving a DRM format from an external API, often the 104 application must use information from the external API to accurately map the 105 DRM format to a elink:VkFormat. 106 For example, DRM formats do not distinguish between RGB and sRGB (as of 107 2018-03-28); external information is required to identify the image's 108 colorspace. 109 110 The mapping between elink:VkFormat and DRM format is also incomplete. 111 For some DRM formats there exist no corresponding Vulkan format, and for 112 some Vulkan formats there exist no corresponding DRM format. 113 114 ==== Usage Patterns 115 116 Three primary usage patterns are intended for this extension: 117 118 * *Negotiation.* The application negotiates with _modifier_-aware, 119 external components to determine sets of image creation parameters 120 supported among all components. 121 + 122 -- 123 In the Linux ecosystem, the negotiation usually assumes the image is a 2D, 124 single-sampled, non-mipmapped, non-array image; this extension permits that 125 assumption but does not require it. 126 The result of the negotiation usually resembles a set of tuples such as 127 _(drmFormat, drmFormatModifier)_, where each participating component 128 supports all tuples in the set. 129 130 Many details of this negotiation—such as the protocol used during 131 negotiation, the set of image creation parameters expressable in the 132 protocol, and how the protocol chooses which process and which API will 133 create the image—are outside the scope of this specification. 134 135 In this extension, flink:vkGetPhysicalDeviceFormatProperties2 with 136 slink:VkDrmFormatModifierPropertiesListEXT serves a primary role during the 137 negotiation, and flink:vkGetPhysicalDeviceImageFormatProperties2 with 138 slink:VkPhysicalDeviceImageDrmFormatModifierInfoEXT serves a secondary role. 139 -- 140 141 * *Import.* The application imports an image with a _modifier_. 142 + 143 -- 144 In this pattern, the application receives from an external source the 145 image's memory and its creation parameters, which are often the result of 146 the negotiation described above. 147 Some image creation parameters are implicitly defined by the external 148 source; for example, ename:VK_IMAGE_TYPE_2D is often assumed. 149 Some image creation parameters are usually explicit, such as the image's 150 pname:format, pname:drmFormatModifier, and pname:extent; and each plane's 151 pname:offset and pname:rowPitch. 152 153 Before creating the image, the application first verifies that the physical 154 device supports the received creation parameters by querying 155 flink:vkGetPhysicalDeviceFormatProperties2 with 156 slink:VkDrmFormatModifierPropertiesListEXT and 157 flink:vkGetPhysicalDeviceImageFormatProperties2 with 158 slink:VkPhysicalDeviceImageDrmFormatModifierInfoEXT. 159 Then the application creates the image by chaining 160 slink:VkImageDrmFormatModifierExplicitCreateInfoEXT and 161 slink:VkExternalMemoryImageCreateInfo onto slink:VkImageCreateInfo. 162 -- 163 164 * *Export.* The application creates an image and allocates its memory. 165 Then the application exports to _modifier_-aware consumers the image's 166 memory handles; its creation parameters; its _modifier_; and the 167 <<VkSubresourceLayout,pname:offset>>, 168 <<VkSubresourceLayout,pname:size>>, and 169 <<VkSubresourceLayout,pname:rowPitch>> of each _memory plane_. 170 + 171 -- 172 In this pattern, the Vulkan device is the authority for the image; it is the 173 allocator of the image's memory and the decider of the image's creation 174 parameters. 175 When choosing the image's creation parameters, the application usually 176 chooses a tuple _(format, drmFormatModifier)_ from the result of the 177 negotiation described above. 178 The negotiation's result often contains multiple tuples that share the same 179 format but differ in their _modifier_. 180 In this case, the application should defer the choice of the image's 181 _modifier_ to the Vulkan implementation by providing all such _modifiers_ to 182 slink:VkImageDrmFormatModifierListCreateInfoEXT::pname:pDrmFormatModifiers; 183 and the implementation should choose from pname:pDrmFormatModifiers the 184 optimal _modifier_ in consideration with the other image parameters. 185 186 The application creates the image by chaining 187 slink:VkImageDrmFormatModifierListCreateInfoEXT and 188 slink:VkExternalMemoryImageCreateInfo onto slink:VkImageCreateInfo. 189 The protocol and APIs by which the application will share the image with 190 external consumers will likely determine the value of 191 slink:VkExternalMemoryImageCreateInfo::pname:handleTypes. 192 The implementation chooses for the image an optimal _modifier_ from 193 slink:VkImageDrmFormatModifierListCreateInfoEXT::pname:pDrmFormatModifiers. 194 The application then queries the implementation-chosen _modifier_ with 195 flink:vkGetImageDrmFormatModifierPropertiesEXT, and queries the memory 196 layout of each plane with flink:vkGetImageSubresourceLayout. 197 198 The application then allocates the image's memory with 199 slink:VkMemoryAllocateInfo, adding chained extension structures for external 200 memory; binds it to the image; and exports the memory, for example, with 201 flink:vkGetMemoryFdKHR. 202 203 Finally, the application sends the image's creation parameters, its 204 _modifier_, its per-plane memory layout, and the exported memory handle to 205 the external consumers. 206 The details of how the application transmits this information to external 207 consumers is outside the scope of this specification. 208 -- 209 210 ==== Prior Art 211 212 Extension 213 `EGL_EXT_image_dma_buf_import`^<<VK_EXT_image_drm_format_modifier-fn1,1>>^ 214 introduced the ability to create an code:EGLImage by importing for each 215 plane a dma_buf, offset, and row pitch. 216 217 Later, extension 218 `EGL_EXT_image_dma_buf_import_modifiers`^<<VK_EXT_image_drm_format_modifier-fn2,2>>^ 219 introduced the ability to query which combination of formats and _modifiers_ 220 the implementation supports and to specify _modifiers_ during creation of 221 the code:EGLImage. 222 223 Extension 224 `EGL_MESA_image_dma_buf_export`^<<VK_EXT_image_drm_format_modifier-fn3,3>>^ 225 is the inverse of `EGL_EXT_image_dma_buf_import_modifiers`. 226 227 The Linux kernel modesetting API (KMS), when configuring the display's 228 framebuffer with `struct 229 drm_mode_fb_cmd2`^<<VK_EXT_image_drm_format_modifier-fn4,4>>^, allows one to 230 specify the frambuffer's _modifier_ as well as a per-plane memory handle, 231 offset, and row pitch. 232 233 GBM, a graphics buffer manager for Linux, allows creation of a `gbm_bo` 234 (that is, a graphics _buffer object_) by importing data similar to that in 235 `EGL_EXT_image_dma_buf_import_modifiers`^<<VK_EXT_image_drm_format_modifier-fn1,1>>^; 236 and symmetrically allows exporting the same data from the `gbm_bo`. 237 See the references to _modifier_ and _plane_ in 238 `gbm.h`^<<VK_EXT_image_drm_format_modifier-fn5,5>>^. 239 240 241 === New Object Types 242 243 None. 244 245 === New Enum Constants 246 247 * Extending elink:VkResult: 248 ** ename:VK_ERROR_INVALID_DRM_FORMAT_MODIFIER_PLANE_LAYOUT_EXT 249 250 * Extending elink:VkStructureType: 251 ** ename:VK_STRUCTURE_TYPE_DRM_FORMAT_MODIFIER_PROPERTIES_LIST_EXT 252 ** ename:VK_STRUCTURE_TYPE_DRM_FORMAT_MODIFIER_PROPERTIES_EXT 253 ** ename:VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_IMAGE_DRM_FORMAT_MODIFIER_INFO_EXT 254 ** ename:VK_STRUCTURE_TYPE_IMAGE_DRM_FORMAT_MODIFIER_LIST_CREATE_INFO_EXT 255 ** ename:VK_STRUCTURE_TYPE_IMAGE_DRM_FORMAT_MODIFIER_EXPLICIT_CREATE_INFO_EXT 256 ** ename:VK_STRUCTURE_TYPE_IMAGE_DRM_FORMAT_MODIFIER_PROPERTIES_EXT 257 258 * Extending elink:VkImageTiling: 259 ** ename:VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT 260 261 * Extending elink:VkImageAspectFlagBits: 262 ** ename:VK_IMAGE_ASPECT_MEMORY_PLANE_0_BIT_EXT 263 ** ename:VK_IMAGE_ASPECT_MEMORY_PLANE_1_BIT_EXT 264 ** ename:VK_IMAGE_ASPECT_MEMORY_PLANE_2_BIT_EXT 265 ** ename:VK_IMAGE_ASPECT_MEMORY_PLANE_3_BIT_EXT 266 267 === New Enums 268 269 None. 270 271 === New Structures 272 273 * Extends slink:VkFormatProperties2: 274 ** slink:VkDrmFormatModifierPropertiesListEXT 275 276 * Member of slink:VkDrmFormatModifierPropertiesListEXT: 277 ** slink:VkDrmFormatModifierPropertiesEXT 278 279 * Extends slink:VkPhysicalDeviceImageFormatInfo2: 280 ** slink:VkPhysicalDeviceImageDrmFormatModifierInfoEXT 281 282 * Extends slink:VkImageCreateInfo: 283 ** slink:VkImageDrmFormatModifierListCreateInfoEXT 284 ** slink:VkImageDrmFormatModifierExplicitCreateInfoEXT 285 286 * Parameter to flink:vkGetImageDrmFormatModifierPropertiesEXT: 287 ** slink:VkImageDrmFormatModifierPropertiesEXT 288 289 === New Functions 290 291 * flink:vkGetImageDrmFormatModifierPropertiesEXT 292 293 === Issues 294 295 1) Should this extension define a single DRM format modifier per 296 sname:VkImage? Or define one per plane? 297 + 298 -- 299 *RESOLVED*: There exists a single DRM format modifier per sname:VkImage. 300 301 *DISCUSSION*: Prior art, such as 302 `EGL_EXT_image_dma_buf_import_modifiers`^<<VK_EXT_image_drm_format_modifier-fn2,2>>^, 303 `struct drm_mode_fb_cmd2`^<<VK_EXT_image_drm_format_modifier-fn4,4>>^, and 304 `struct 305 gbm_import_fd_modifier_data`^<<VK_EXT_image_drm_format_modifier-fn5,5>>^, 306 allows defining one _modifier_ per plane. 307 However, developers of the GBM and kernel APIs concede it was a mistake. 308 Beginning in Linux 4.10, the kernel requires that the application provide 309 the same DRM format _modifier_ for each plane. 310 (See Linux commit 311 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=bae781b259269590109e8a4a8227331362b88212[bae781b259269590109e8a4a8227331362b88212]). 312 And GBM provides an entrypoint, code:gbm_bo_get_modifier, for querying the 313 _modifier_ of the image but does not provide one to query the modifier of 314 individual planes. 315 -- 316 317 2) When creating an image with 318 slink:VkImageDrmFormatModifierExplicitCreateInfoEXT, which is typically 319 used when _importing_ an image, should the application explicitly provide 320 the size of each plane? 321 + 322 -- 323 *RESOLVED*: No. 324 The application must: not provide the size. 325 To enforce this, the API requires that 326 slink:VkImageDrmFormatModifierExplicitCreateInfoEXT::pname:pPlaneLayouts::<<VkSubresourceLayout,pname:size>> 327 must: be 0. 328 329 *DISCUSSION*: Prior art, such as 330 `EGL_EXT_image_dma_buf_import_modifiers`^<<VK_EXT_image_drm_format_modifier-fn2,2>>^, 331 `struct drm_mode_fb_cmd2`^<<VK_EXT_image_drm_format_modifier-fn4,4>>^, and 332 `struct 333 gbm_import_fd_modifier_data`^<<VK_EXT_image_drm_format_modifier-fn5,5>>^, 334 omits from the API the size of each plane. 335 Instead, the APIs infer each plane's size from the import parameters, which 336 include the image's pixel format and a dma_buf, offset, and row pitch for 337 each plane. 338 339 However, Vulkan differs from EGL and GBM with regards to image creation in 340 the following ways: 341 342 .Differences in Image Creation 343 344 - *Undedicated allocation by default.* When importing or exporting a set 345 of dma_bufs as an code:EGLImage or code:gbm_bo, common practice mandates 346 that each dma_buf's memory be dedicated (in the sense of 347 `VK_KHR_dedicated_allocation`) to the image (though not necessarily 348 dedicated to a single plane). 349 In particular, neither the GBM documentation nor the EGL extension 350 specifications explicitly state this requirement, but in light of common 351 practice this is likely due to under-specification rather than 352 intentional omission. 353 In contrast, `VK_EXT_image_drm_format_modifier` permits, but does not 354 require, the implementation to require dedicated allocations for images 355 created with ename:VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT. 356 357 - *Separation of image creation and memory allocation.* When importing a 358 set of dma_bufs as an code:EGLImage or code:gbm_bo, EGL and GBM create 359 the image resource and bind it to memory (the dma_bufs) simultaneously. 360 This allows EGL and GBM to query each dma_buf's size during image 361 creation. 362 In Vulkan, image creation and memory allocation are independent unless a 363 dedicated allocation is used (as in `VK_KHR_dedicated_allocation`). 364 Therefore, without requiring dedicated allocation, Vulkan cannot query 365 the size of each dma_buf (or other external handle) when calculating the 366 image's memory layout. 367 Even if dedication allocation were required, Vulkan cannot calculate the 368 image's memory layout until after the image is bound to its dma_ufs. 369 370 The above differences complicate the potential inference of plane size in 371 Vulkan. 372 Consider the following problematic cases: 373 374 .Problematic Plane Size Calculations 375 376 - *Padding.* Some plane of the image may require implementation-dependent 377 padding. 378 379 - *Metadata.* For some _modifiers_, the image may have a metadata plane 380 which requires a non-trivial calculation to determine its size. 381 382 - *Mipmapped, array, and 3D images.* The implementation may support 383 ename:VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT for images whose 384 pname:mipLevels, pname:arrayLayers, or pname:depth is greater than 1. 385 For such images with certain _modifiers_, the calculation of each 386 plane's size may be non-trivial. 387 388 However, an application-provided plane size solves none of the above 389 problems. 390 391 For simplicity, consider an external image with a single memory plane. 392 The implementation is obviously capable calculating the image's size when 393 its tiling is ename:VK_IMAGE_TILING_OPTIMAL. 394 Likewise, any reasonable implementation is capable of calculating the 395 image's size when its tiling uses a supported _modifier_. 396 397 Suppose that the external image's size is smaller than the 398 implementation-calculated size. 399 If the application provided the external image's size to 400 flink:vkCreateImage, the implementation would observe the mismatched size 401 and recognize its inability to comprehend the external image's layout 402 (unless the implementation used the application-provided size to select a 403 refinement of the tiling layout indicated by the _modifier_, which is 404 strongly discouraged). 405 The implementation would observe the conflict, and reject image creation 406 with ename:VK_ERROR_INVALID_DRM_FORMAT_MODIFIER_PLANE_LAYOUT_EXT. 407 On the other hand, if the application did not provide the external image's 408 size to flink:vkCreateImage, then the application would observe after 409 calling flink:vkGetImageMemoryRequirements that the external image's size is 410 less than the size required by the implementation. 411 The application would observe the conflict and refuse to bind the 412 sname:VkImage to the external memory. 413 In both cases, the result is explicit failure. 414 415 Suppose that the external image's size is larger than the 416 implementation-calculated size. 417 If the application provided the external image's size to 418 flink:vkCreateImage, for reasons similar to above the implementation would 419 observe the mismatched size and recognize its inability to comprehend the 420 image data residing in the extra size. 421 The implementation, however, must assume that image data resides in the 422 entire size provided by the application. 423 The implementation would observe the conflict and reject image creation with 424 ename:VK_ERROR_INVALID_DRM_FORMAT_MODIFIER_PLANE_LAYOUT_EXT. 425 On the other hand, if the application did not provide the external image's 426 size to flink:vkCreateImage, then the application would observe after 427 calling flink:vkGetImageMemoryRequirements that the external image's size is 428 larger than the implementation-usable size. 429 The application would observe the conflict and refuse to bind the 430 sname:VkImage to the external memory. 431 In both cases, the result is explicit failure. 432 433 Therefore, an application-provided size provides no benefit, and this 434 extension should not require it. 435 This decision renders slink:VkSubresourceLayout::pname:size an unused field 436 during image creation, and thus introduces a risk that implementations may 437 require applications to submit sideband creation parameters in the unused 438 field. 439 To prevent implementations from relying on sideband data, this extension 440 _requires_ the application to set pname:size to 0. 441 -- 442 443 ==== References 444 445 . [[VK_EXT_image_drm_format_modifier-fn1]] 446 https://www.khronos.org/registry/EGL/extensions/EXT/EGL_EXT_image_dma_buf_import.txt[`EGL_EXT_image_dma_buf_import`] 447 . [[VK_EXT_image_drm_format_modifier-fn2]] 448 https://www.khronos.org/registry/EGL/extensions/EXT/EGL_EXT_image_dma_buf_import_modifiers.txt[`EGL_EXT_image_dma_buf_import_modifiers`] 449 . [[VK_EXT_image_drm_format_modifier-fn3]] 450 https://www.khronos.org/registry/EGL/extensions/MESA/EGL_MESA_image_dma_buf_export.txt[`EGL_MESA_image_dma_buf_export`] 451 . [[VK_EXT_image_drm_format_modifier-fn4]] 452 https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/include/uapi/drm/drm_mode.h?id=refs/tags/v4.10#n392[`struct 453 drm_mode_fb_cmd2`] 454 . [[VK_EXT_image_drm_format_modifier-fn5]] 455 https://cgit.freedesktop.org/mesa/mesa/tree/src/gbm/main/gbm.h?id=refs/tags/mesa-18.0.0-rc1[`gbm.h`] 456 457 ==== Version History 458 459 * Revision 1.0, 2018-08-29 (Chad Versace) 460 - First stable revision