Skip to content

OutOfMemoryError (Out of stack space) in SpriteUpdateUtil.transitionLayouts with large modded resource pack atlas (8192x4096) #880

Description

@clayhuey77-blip

Summary: VulkanMod 0.6.1 on Minecraft 1.21.11 crashes with OutOfMemoryError: Out of stack space when connecting to a server whose resource pack produces a very large item atlas (8192x4096 in this case). The crash originates in SpriteUpdateUtil.transitionLayouts via VulkanImage.transitionLayout's MemoryStack allocation.

Steps to reproduce:

Install VulkanMod 0.6.1 on Fabric 1.21.11
Connect to a server with a large custom resource pack (item atlas builds to 8192x4096 in this case)
Game disconnects immediately after the atlas is built, citing a resource pack load failure

Tested and ruled out:

-Xss8M (JVM thread stack) — no effect
-Dorg.lwjgl.system.stackSize=8192 (LWJGL native stack) — no effect
Disabling "Texture Animations" in VulkanMod's own settings — no effect (crash occurs before this setting's code path is even reached)

Stack trace:

java.lang.OutOfMemoryError: Out of stack space.
at org.lwjgl.system.MemoryStack.nmalloc(MemoryStack.java:322)
at org.lwjgl.system.MemoryStack.ncalloc(MemoryStack.java:340)
at org.lwjgl.vulkan.VkImageMemoryBarrier.calloc(VkImageMemoryBarrier.java:426)
at net.vulkanmod.vulkan.texture.VulkanImage.transitionLayout(VulkanImage.java:368)
at net.vulkanmod.vulkan.texture.VulkanImage.transitionLayout(VulkanImage.java:361)
at net.vulkanmod.vulkan.texture.VulkanImage.transitionImageLayout(VulkanImage.java:355)
at net.vulkanmod.vulkan.texture.VulkanImage.transitionImageLayout(VulkanImage.java:290)
at net.vulkanmod.vulkan.texture.VulkanImage.readOnlyLayout(VulkanImage.java:282)
at net.vulkanmod.render.texture.SpriteUpdateUtil.lambda$transitionLayouts$0(SpriteUpdateUtil.java:35)
at java.lang.Iterable.forEach(Unknown Source)
at net.vulkanmod.render.texture.SpriteUpdateUtil.transitionLayouts(SpriteUpdateUtil.java:35)
at net.vulkanmod.render.texture.ImageUploadHelper.submitCommands(ImageUploadHelper.java:24)
at net.vulkanmod.vulkan.memory.buffer.StagingBuffer.submitUploads(StagingBuffer.java:61)
at net.vulkanmod.vulkan.memory.buffer.StagingBuffer.copyBuffer(StagingBuffer.java:38)
at net.vulkanmod.vulkan.texture.VulkanImage.uploadSubTextureAsync(VulkanImage.java:244)
at net.vulkanmod.vulkan.texture.VTextureSelector.uploadSubTexture(VTextureSelector.java:67)
at net.vulkanmod.render.engine.VkCommandEncoder.writeToTexture(VkCommandEncoder.java:439)

System info:

VulkanMod 0.6.1, Minecraft 1.21.11, Fabric Loader 0.19.3, Java 21
Windows, LWJGL 3.3.3
Atlas sizes at crash time: blocks.png-atlas 4096x2048x4, items.png-atlas 8192x4096x0

Suspected cause: transitionLayouts appears to allocate a VkImageMemoryBarrier per sprite via MemoryStack.calloc inside a single unflushed stack frame/forEach pass. With thousands of sprites in an oversized atlas, this overflows the stack regardless of -Xss or LWJGL's native stack size setting, suggesting the allocations aren't being popped/released between iterations.

Full latest.log attached.

Server attempted to connect to: OriginRealms.com

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions