Conversation
A ConnectX-6 with PF0 in SR-IOV mode and PF1 dedicated is advertised as two independent components. No broker change was needed: model matching is by string, SmartNIC port matching is by local_name and so port-count agnostic, and components are excluded by name. These tests pin that behaviour down. Covers: the dedicated half allocating its single port; a two-port ConnectX-6 request not matching the single-port ConnectX-6-1P card; allocating either half leaving the other untouched; and exhausting every VF of PF0 having no effect on PF1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016wB1CV7hRtwmWVLPY8GBxh
__exclude_allocated_pci_device_from_shared_nic decremented the delegated capacity with `delegated_capacity -= allocated.get_capacity_allocations()`. Capacities defines __sub__ but not __isub__, so that built a new object and rebound the local while the delegation kept its original unit count - even though the BDF list beside it was being mutated in place. A Shared NIC with every BDF allocated was therefore never excluded from the candidate node, because the capacity check still reported the original units. The next allocation reached usable_bdfs[0] on an empty list and raised IndexError instead of INSUFFICIENT_RESOURCES. The comment at the capacity check - "Delegated capacity would have been decremented already to exclude allocated shared NICs" - shows the decrement was meant to persist. The remaining BDFs are the source of truth, so derive the unit count from them and mutate the delegated object in place, the same way the BDF list is already updated. Also guards a missing or empty label delegation, which previously raised AttributeError. Four of the tests in test_mixed_mode_nic.py fail against the old code and pass against this. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016wB1CV7hRtwmWVLPY8GBxh
Lets the mixed-mode NIC branches be deployed together on dev. REVERT THIS COMMIT before merge - main should not depend on a pre-release. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016wB1CV7hRtwmWVLPY8GBxh
Pre-release so the mixed-mode NIC branches can be deployed and tested on dev together. 2.0.1 is not yet on PyPI, so 2.0.1b1 sorts ahead of the released 2.0.0 and behind the eventual 2.0.1. Local image tags follow the version. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016wB1CV7hRtwmWVLPY8GBxh
Matches the AMHandlers pre-release carrying the PCI rescan scoping fix. fabric-am-handlers 2.0.1 is already on PyPI, so the pre-release has to be 2.0.2b1 rather than 2.0.1b1 to sort after it. REVERT THIS COMMIT before merge - main should not build against a pre-release. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016wB1CV7hRtwmWVLPY8GBxh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
A ConnectX-6 can now run one port in SR-IOV/shared mode and the sibling port dedicated. The dedicated half is advertised as a separate component with model
ConnectX-6-1P.1. No broker change was needed for mixed-mode cards
Verified by reading, now pinned down by tests:
__check_component_labels_and_capacitiesmatches on model string equality.__update_smart_nic_labels_and_capacitiesmatches ports bylocal_name, so it is port-count agnostic.__exclude_allocated_componentremoves by component name, so the two halves of the card are already independent.get_matching_nodes_with_componentscounts by(Type, Model)andAggregatedBQMPlugin.occupied_node_capacitybuckets by(resource_type, resource_model), so the new model gets its own capacity/allocated figures for free.2. Bug fix: Shared NIC delegated capacity never decremented
Found while writing those tests, and fixed here.
__exclude_allocated_pci_device_from_shared_nicdid:Capacitiesdefines__sub__but not__isub__, so this builds a new object and rebinds the local — the delegation keeps its original unit count, while the BDF list beside it is mutated in place.Consequence: a Shared NIC with every BDF allocated was never excluded from the candidate node, because the capacity check still reported the original units. The next allocation then reached
usable_bdfs[0]on an empty list and raisedIndexErrorinstead of a cleanINSUFFICIENT_RESOURCES. The comment at the capacity check — "Delegated capacity would have been decremented already to exclude allocated shared NICs" — shows the decrement was meant to persist.The remaining BDFs are the source of truth, so the fix derives the unit count from them and mutates the delegated object in place, exactly the way the BDF list is already updated. Also guards a missing or empty label delegation, which previously raised
AttributeError.This bug predates the mixed-mode work and is independent of it.
Tests
New
fabric_cf/actor/test/unit/test_mixed_mode_nic.py, 9 cases, no infrastructure required.Mixed-mode:
local_nameConnectX-6request does not match the single-portConnectX-6-1PcardCapacity tracking:
INSUFFICIENT_RESOURCESrather thanIndexError4 of these fail against the old code and pass against this branch. The full
fabric_cf/actor/test/unit/directory is green (39 passed).Version
Bumped to 2.0.1b1 for dev deployment. 2.0.1 is not yet on PyPI, so
2.0.1b1sorts ahead of the released 2.0.0 and behind the eventual 2.0.1. Local image tags in the threedocker-compose.ymlfiles follow the version.DEV ONLY: pin fabric-fim to 2.0.1b1DEV ONLY: pin fabric-am-handlers to 2.0.2b1 in Dockerfile-authBoth exist so the branches can be deployed together on dev.
mainshould not depend on or build against pre-releases — drop both before merging.Depends on
fabric-testbed/InformationModel#223
🤖 Generated with Claude Code
https://claude.ai/code/session_016wB1CV7hRtwmWVLPY8GBxh