YARN-11981: Fix 0% capacity for auto created queue - #8674
Open
Hean-Chhinling wants to merge 9 commits into
Open
Conversation
|
🎊 +1 overall
This message was automatically generated. |
Contributor
|
Thanks @Hean-Chhinling for fixing this bug! If i see correctly you checked in the
field on the PR, but i dont see this in github. Do i miss something? |
K0K0V0K
approved these changes
Aug 12, 2026
K0K0V0K
left a comment
K0K0V0K
left a comment
Contributor
There was a problem hiding this comment.
Thanks @Hean-Chhinling !
LGTM!
Contributor
Author
Thanks you for the review @K0K0V0K. |
Contributor
Author
|
Hello @brumi1024 and @slfan1989, |
Member
|
Scanned with an LLM and posted the selected feedback below DISCLAIMER: I reviewed each finding before posting, they look reasonable to me literally, but I'm not an expert in the frontend area, please re-evaluate each item carefully (you can reject the item with reasons if you think it's wrong). Findings
|
brumi1024
requested changes
Aug 19, 2026
Comment on lines
+75
to
+76
| * - flexible v2 (`auto-queue-creation-v2.enabled`) → leaf-template | ||
| * - legacy (`auto-create-child-queue.enabled`) → leaf-queue-template |
Member
There was a problem hiding this comment.
I agree with @pan3793's comment. This doesn't seem to handle parent-template, template and wildcard scenarios. See the docs about flexible AQC:
Contributor
Author
There was a problem hiding this comment.
Thank you for the review @brumi1024!
I purposely not handle the others because in this PR, it is for fixing the bug that the auto leaf-queue capacity is not showing correctly. Therefore, this is to check only for the leaf templates.
Member
There was a problem hiding this comment.
The issue is that we cannot solely rely on the leaf-template. AQCv2's template handling logic takes multiple values into account (template, wildcards, leaf-template), and this change assumes that nothing else matters. See in my comment below: #8674 (comment)
Member
|
Thanks for the fix @Hean-Chhinling, the bug report is valid - the card really does show 0% for a queue that the RM is giving 3w. But I'd like to suggest a different approach, because the current one will keep missing cases. The root cause is where the card gets its number. This PR works around that by rebuilding the template path in the UI (.auto-queue-creation-v2.leaf-template). The problem is that the RM's template resolution is more involved than that one key: there are template, leaf-template and parent-template with a precedence order, wildcard templates like root.default.*.auto-queue-creation-v2.leaf-template.capacity for deeper queues, dynamic parent queues that use parent-template, and a default of 1w when no template is configured at all (AbstractCSQueue.setupConfigurableCapacities). Reimplementing all of that in TypeScript means a second copy of the logic that will drift from the RM, the main thing I wanted to avoid in the config UI. The good news is we don't have to: the /ws/v1/cluster/scheduler response already contains the RM-computed answer for every queue, and the UI already parses it into QueueInfo - look at weight, normalizedWeight and queueCapacityVectorInfo.configuredCapacityVector in types/queue.ts. They're just not used by the card yet. So the suggestion: in transformToCardData, when creationMethod is dynamicFlexible or dynamicLegacy, take capacity / max capacity from those REST fields, and only let a staged config value (isStaged === true) override it, so edits still show before apply. That removes the need for the new template-path helper entirely, and it fixes wildcard, parent-template and the no-template default in one go. |
Hean-Chhinling
changed the title
Hean-Chhinling
force-pushed
the
trunk_YARN-11981
branch
from
August 19, 2026 16:49
3596555 to
09a99a3
Compare
Hean-Chhinling
force-pushed
the
trunk_YARN-11981
branch
from
August 19, 2026 16:50
09a99a3 to
f5d6319
Compare
Contributor
Author
|
Thank you so much @pan3793 and @brumi1024 for the review! |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
brumi1024
reviewed
Aug 24, 2026
Hean-Chhinling
force-pushed
the
trunk_YARN-11981
branch
from
August 24, 2026 15:57
123d700 to
4d8b820
Compare
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
brumi1024
reviewed
Aug 24, 2026
| configuredMinResource && | ||
| (configuredMinResource.memory > 0 || configuredMinResource.vCores > 0) | ||
| ) { | ||
| capacityConfig = `[memory-mb=${configuredMinResource.memory},vcores=${configuredMinResource.vCores}]`; |
Member
There was a problem hiding this comment.
The scheduler REST response already provides queueCapacityVectorInfo.configuredCapacityVector. Reconstructing the value from weight, configuredMinResource, and the computed capacity is lossy. For example, a valid mixed vector such as [memory-mb=16384,vcores=100%] has weight = -1 and may have configuredMinResource = 0/0, so this code displays the computed scalar capacity instead. It also drops custom resources (like GPU/FPGA/etc). Could we type and use configuredCapacityVector directly?
The general idea is the less custom logic on the UI the better, we should rely on the RM as much as possible, so that if something changes we don't need to constantly update the UI logic as well.
Contributor
Author
There was a problem hiding this comment.
Thanks for actively reviewing the PR @brumi1024!
I did not use configuredCapacityVector intentionally because when I checked the /scheduler endpoint it shows as the following despite I configured the auto queue created capacity to (2048,2):
<queueCapacityVectorInfo>
<configuredCapacityVector>[memory-mb=0.0%,vcores=0.0%]</configuredCapacityVector>
</queueCapacityVectorInfo>
And at configuredMinResource it shows correctly, therefore I used it instead:
<configuredMinResource>
<memory>2048</memory>
<vCores>2</vCores>
</configuredMinResource>
Shouldn't the configuredMinResource is the right one here?
Member
There was a problem hiding this comment.
For the legacy queue mode? Yes. For the non-legacy one? We should show what is provided. And this logic omits any custom resources we might have:
capacityConfig = [memory-mb=${configuredMinResource.memory},vcores=${configuredMinResource.vCores}];
It was a while ago I touched the UI, but I think there should already be a logic that parses and shows the resources from a resource object.
Contributor
Author
There was a problem hiding this comment.
Thanks! I am able to test out successfully once I triggered the queue creation in mixed mode.
Now it shows correctly in the UI:
Except for the MaxCapacity, I am not sure how would it be possible to handle mixedMode within /scheduler endpoint. Because the maxCapacityVector is not exist anywhere there.
Member
There was a problem hiding this comment.
Yes, this is flexible queue mode, but legacy AQC, I was referring to the flexible AQC.
What about something like this? brumi1024@7ae0528
With this we wouldn't require parsing the config again on the UI.
Contributor
Author
There was a problem hiding this comment.
Thank you so much for the help, @brumi1024!
That is really nice approach to handle the logic at /store instead of /hooks. Also exposing the maxCapacityVector at the backend.
I have applied your patch, added a unit test then did a manual testing and all seems to be working as expected:
<queueCapacitiesByPartition>
<partitionName/>
<queueCapacityVectorInfo>
<configuredCapacityVector>[memory-mb=2048.0,vcores=2.0]</configuredCapacityVector>
<capacityVectorEntries>
<resourceName>memory-mb</resourceName>
<resourceValue>2048.0</resourceValue>
</capacityVectorEntries>
<capacityVectorEntries>
<resourceName>vcores</resourceName>
<resourceValue>2.0</resourceValue>
</capacityVectorEntries>
</queueCapacityVectorInfo>
<maximumQueueCapacityVectorInfo>
<configuredCapacityVector>[memory-mb=4096.0,vcores=4.0]</configuredCapacityVector>
<capacityVectorEntries>
<resourceName>memory-mb</resourceName>
<resourceValue>4096.0</resourceValue>
</capacityVectorEntries>
<capacityVectorEntries>
<resourceName>vcores</resourceName>
<resourceValue>4.0</resourceValue>
</capacityVectorEntries>
</maximumQueueCapacityVectorInfo>
</queueCapacitiesByPartition>
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
Hean-Chhinling
force-pushed
the
trunk_YARN-11981
branch
from
August 26, 2026 15:46
69e49c4 to
60cf053
Compare
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
… format to include the new maximumQueueCapacityVectorInfo field
|
💔 -1 overall
This message was automatically generated. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Contains content generated by Cursor.
Description of PR
Utilise existing
queueInfoto get the configured capacity for auto created queue once it is turned on.Formatted the same way as static queues (e.g.
3w)Staged capacity edits from "Edit Capacity" still take precedence before apply.
How was this patch tested?
Tested locally and works as expected. Please refer to the screenshot below:
For code changes:
declared according to the connector-specific documentation? Note: Automated CI
testing doesn't cover all cases so manual testing with cloud storage is still
required.
LICENSE,LICENSE-binary,NOTICE-binaryfiles?AI Tooling
If an AI tool was used:
where is the name of the AI tool used.
https://www.apache.org/legal/generative-tooling.html