Fail fast when receiver topology is empty - #104
Closed
koriyoshi2041 wants to merge 1 commit into
Closed
Conversation
Author
|
Closing after a deeper reachability audit. The update path returns non-target ranks before bucket generation, while gathered local topology includes each valid world rank, so an empty selected topology is not reachable for a valid non-empty target list. The helper can loop on a malformed direct input, but that does not justify the core-path claim in this PR. Thanks for the review bandwidth. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Problem
_gen_h2d_buckets()intersects targeted ranks with each participant's local topology before calling_assign_receiver_ranks(). A participant with no selected local rank can therefore reach receiver assignment with non-empty global buckets and an empty local topology.In that state,
receiver_listis empty whileflattened_bucketsis not. The outer assignment loop never advancesassigned_cnt, so the update hangs indefinitely.An isolated CPU reproduction calling
_assign_receiver_ranks([(7, "bucket")], {}, {"mlx5_0": {7}})remained alive after 750 ms and had to be terminated.Fix
Raise a clear
ValueErrorbefore entering the assignment loop when no local receiver can be selected. Returning an empty assignment would risk turning the collective mismatch into a false-success update.Testing