fix: degrade empty anchor sampling to a masked batch to avoid multi-r… - #737
Open
Boreas618 wants to merge 2 commits into
Open
fix: degrade empty anchor sampling to a masked batch to avoid multi-r…#737Boreas618 wants to merge 2 commits into
Boreas618 wants to merge 2 commits into
Conversation
…ank hang Raising when a micro-batch has no valid anchors skips the loss-denominator all-reduce and hangs peer ranks until NCCL timeout. Factor candidate selection into `_anchor_candidates` so DSpark can override eligibility without duplicating the sampler.
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
Collaborator
|
Overall LGTM, please rebase and pass CI. |
…ulti-rank-hang # Conflicts: # specforge/algorithms/common/dflash_family_model.py # tests/test_utils/test_dflash_losses.py
jiapingW
self-requested a review
August 18, 2026 02:44
Collaborator
|
@Boreas618 @maocheng23 I have fixed the conflicts. |
Collaborator
|
help cc. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
When a micro-batch has no valid anchor positions,
_sample_anchor_positionsused to raise. In multi-rank training that exit happens before the loss-denominator all-reduce, so peer ranks block on the collective until NCCL timeout.This shows up when sanitization or sparse supervision leaves a rank with an empty candidate set (including DSpark cases where tokens exist but no adjacent supervised pair). The sampler should still participate in the collective instead of aborting the step.
Modifications
_anchor_candidates(eligibility / width) and_select_anchors(random pick); mark the shared path@final.keep_maskallFalse) that contributes zero supervision but still reaches the collective.@overrideof_anchor_candidates; drop the duplicated Domino sampler override so it inherits the shared path.Related Issues
N/A
Accuracy Test
N/A — training control-flow only; empty batches contribute zero loss rather than changing the model or loss formulation for valid anchors.
Benchmark & Profiling
N/A — no expected throughput/latency impact on the happy path; only the rare empty-candidate case changes from hang/raise to a no-op step.
Checklist