Skip to content

gfxstream: answer a deferred AHB image layout from the AHB itself - #176

Open
jvle wants to merge 2 commits into
google:mainfrom
jvle:deferred-ahb-image-layout
Open

gfxstream: answer a deferred AHB image layout from the AHB itself#176
jvle wants to merge 2 commits into
google:mainfrom
jvle:deferred-ahb-image-layout

Conversation

@jvle

@jvle jvle commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Second of the pieces @gurchetansingh asked for on the ARSP review — the hand-coded
half. The regenerated vk_decoder.cpp and the mappable-only-blob change follow as
separate PRs.

What goes wrong

Intel ANV leaves an AHB-external image's layout unresolved until vkBindImageMemory
and reports size=0, alignment=0 until then. That is spec-valid, but guests take it
literally:

MESA: error: Failed to allocate coherent memory: failed to allocate on the host: -1.
MESA: error: zink: couldn't allocate memory: heap=0 size=0
weston.service: Main process exited, code=killed, status=11/SEGV

The guest allocates and maps host-visible memory in one step inside
vkAllocateMemory (ResourceTracker::getCoherentMemory), so a zero-sized request
fails at the host allocation with VK_ERROR_OUT_OF_HOST_MEMORY — the -1 above —
before anything is mapped.

Why the image is AHB-external at all

Not the guest's doing. A Linux guest asks for DMA_BUF_BIT_EXT, and
transformImpl_VkImageCreateInfo_tohost ORs in the host's default handle type, which
is AHB on an Android host. That is deliberate — it is what lets the guest's
window-system image alias an AHB-backed ColorBuffer — so the requirements have to be
answered rather than the handle type avoided.

I did try the other direction (dmabuf end to end via VulkanExternalMemoryMode:OpaqueFd)
and it does remove this failure, but scanout then breaks: the Android display path needs
AHB for zero-copy, and the composited frame never reaches the surface. So this seemed
like the right place to fix it.

Approach

Allocate the AHB up front for an image the driver refuses to describe, take the real
size from vkGetAndroidHardwareBufferPropertiesANDROID, substitute it, and import the
same AHB on the guest's dedicated allocation so the driver resolves the layout on bind.

Placement is load-bearing: after updateImageMemorySizeLocked (which rewrites the
struct for compressed images) and before transformToGuestMemoryRequirements, because
ahbProps.memoryTypeBits is in host indices and that transform is what maps them to
guest indices.

Note on scope

on_vkGetImageSubresourceLayout is added here but is not reached until the decoder
routes that call through VkDecoderGlobalState — that is
Mesa MR 43969 plus the
regenerated vk_decoder.cpp, which I will send once the Mesa side lands.

Testing

Intel PTL Android host: Weston reaches GL renderer: zink Vulkan 1.4 (Intel(R) Graphics (PTL)), the host substitutes size=3768320 for the 1280x720 scanout image, and
glmark2-wayland runs accelerated on zink/Intel PTL.

@gurchetansingh gurchetansingh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you separate the codegen changes to commit (1) [with a no-op impl] and the actual implemntation to commit (2)? Thanks!

// AHB is imported at the dedicated allocation so the driver resolves the layout on bind.
// shared_ptr with a releasing deleter so every mImageInfo teardown path frees it, including
// clearLocked()'s bulk clear() which runs neither destroy helper.
std::shared_ptr<AHardwareBuffer> deferredLayoutAhb;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Group these into:

    struct DeferredLayoutInfo {
        std::shared_ptr<AHardwareBuffer> ahb;
        VkDeviceSize size = 0;
        VkDeviceSize alignment = 0;
        uint32_t memoryTypeBits = 0;
        VkDeviceSize rowPitch = 0;
    };

// TODO: might need to use an array of layouts to represent each sub resource
VkImageLayout layout = VK_IMAGE_LAYOUT_UNDEFINED;
VkDeviceMemory memory = VK_NULL_HANDLE;
// External memory handle types the image was created with, taken from the

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Umm, looks like LLM-speak :-), I suggest rewording this or just deleting this.

Comment thread host/vulkan/vk_decoder_global_state.cpp Outdated
return;
}

// Deferred image layout (see ImageInfo::deferredLayoutAhb). Placed AFTER

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be consolidated into a helper function:

    void updateImageMemoryRequirementsLocked(VkDevice device, VkImage image,                                                                           
                                             VkMemoryRequirements* pMemoryRequirements) REQUIRES(mMutex) {                                             
        auto* imageInfo = gfxstream::base::find(mImageInfo, image);                                                                                    
        if (!imageInfo) return;                                                                                                                        
                                                                                                                                                       
        if (imageInfo->compressInfo) {                                                                                                                 
            *pMemoryRequirements = imageInfo->compressInfo->getMemoryRequirements();                                                                   
            return;                                                                                                                                    
        }                                                                                                                                              
    #ifdef __ANDROID__                                                                                                                                 
        if (pMemoryRequirements && pMemoryRequirements->size == 0 && imageInfo->deferredLayoutSize > 0) {                                              
            pMemoryRequirements->size = imageInfo->deferredLayoutSize;                                                                                 
            pMemoryRequirements->alignment = imageInfo->deferredLayoutAlignment;                                                                       
            pMemoryRequirements->memoryTypeBits = imageInfo->deferredLayoutMemoryTypeBits;                                                             
        }                                                                                                                                              
    #endif                                                                                                                                             
    }                                                                                                                                                  

.pNext = nullptr,
.buffer = nullptr,
};
if (dedicatedAllocInfoPtr && dedicatedAllocInfoPtr->image != VK_NULL_HANDLE) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hold a shared_ptr<> since you drop the mutex:

   std::shared_ptr<AHardwareBuffer> deferredAhbHold;                                                                                                  
    if (dedicatedAllocInfoPtr && dedicatedAllocInfoPtr->image != VK_NULL_HANDLE) {                                                                     
        std::lock_guard<std::mutex> dlLock(mMutex);                                                                                                    
        auto* dlInfo = gfxstream::base::find(mImageInfo, dedicatedAllocInfoPtr->image);                                                                
        if (dlInfo && dlInfo->deferredLayoutAhb) {                                                                                                     
            deferredAhbHold = dlInfo->deferredLayoutAhb;                                                                                               
            importDeferredLayoutAhb.buffer = deferredAhbHold.get();                                                                                    

jvle added 2 commits August 25, 2026 16:26
Decode it with emit_global_state_wrapped_decoding, as the other image
entry points already are, so the host can answer it. The implementation
added here just calls through to the driver, so behaviour is unchanged.

The generator side is a separate change in Mesa:
gitlab.freedesktop.org/mesa/mesa/-/merge_requests/43969

vk_decoder.cpp carries only the delta that change produces. A plain
regeneration also rewrites ~87 unrelated lines across three generated
files, because the checked-in output has drifted from the generator;
that is left alone here.

Test: regenerated with and without MR 43969 -- the only difference is
      the dispatch below, and the other generated files are identical
Intel ANV leaves an AHB-external image's layout unresolved until
vkBindImageMemory and reports size=0 until then. That is spec-valid, but
guests take it literally: Mesa zink asks for a zero-sized allocation,
the host rejects it with VK_ERROR_OUT_OF_HOST_MEMORY, and the compositor
dies on the NULL.

The image is AHB-external because gfxstream makes it so -- a Linux guest
asks for DMA_BUF and transformImpl_VkImageCreateInfo_tohost ORs in the
host default, which is what lets the image alias an AHB-backed
ColorBuffer. So the requirements have to be answered, not avoided.

Allocate the AHB up front for an image the driver refuses to describe,
take the size from vkGetAndroidHardwareBufferPropertiesANDROID, and
import the same AHB on the guest's dedicated allocation so the layout
resolves on bind.

The substitution runs in updateImageMemoryRequirementsLocked, so it
happens before transformToGuestMemoryRequirements -- necessary because
ahbProps.memoryTypeBits is in host indices and that transform maps them
to guest indices.

Bug: 545345381
Test: Intel PTL host -- Weston reaches its GL renderer and the host
      substitutes size=3768320 for the 1280x720 scanout image
Test: glmark2-wayland runs accelerated on zink/Intel PTL
@jvle
jvle force-pushed the deferred-ahb-image-layout branch from c2c228e to e181255 Compare August 25, 2026 23:42
@jvle

jvle commented Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

Thanks — all addressed, now two commits.

  • (1) codegen routing + a pass-through impl, no behaviour change. (2) the implementation.
  • DeferredLayoutInfo struct as suggested.
  • Trimmed the comments — you were right, that was too much prose.
  • Both requirements handlers now go through updateImageMemoryRequirementsLocked, which also
    absorbs the old updateImageMemorySizeLocked.
  • shared_ptr held across the lock drop — good catch, that was a real use-after-free.

Retested on Intel PTL: Weston on its GL renderer, substitution and stride both applied, no
transfer or blob errors.

One note on (1): vk_decoder.cpp carries only the delta MR 43969 produces, not a full regen — a
plain regeneration also rewrites ~87 unrelated lines across three generated files, because the
checked-in output has drifted from the generator. Happy to send that separately if you want it
cleaned up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants