Skip to content

Add broker tests for a NIC split between shared and dedicated ports - #454

Open
kthare10 wants to merge 5 commits into
mainfrom
feature/mixed-mode-nic
Open

kthare10 wants to merge 5 commits into
mainfrom
feature/mixed-mode-nic

Conversation

@kthare10

@kthare10 kthare10 commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Context

A ConnectX-6 can now run one port in SR-IOV/shared mode and the sibling port dedicated. The dedicated half is advertised as a separate component with model ConnectX-6-1P.

1. No broker change was needed for mixed-mode cards

Verified by reading, now pinned down by tests:

  • __check_component_labels_and_capacities matches on model string equality.
  • __update_smart_nic_labels_and_capacities matches ports by local_name, so it is port-count agnostic.
  • __exclude_allocated_component removes by component name, so the two halves of the card are already independent.
  • get_matching_nodes_with_components counts by (Type, Model) and AggregatedBQMPlugin.occupied_node_capacity buckets by (resource_type, resource_model), so the new model gets its own capacity/allocated figures for free.

2. Bug fix: Shared NIC delegated capacity never decremented

Found while writing those tests, and fixed here.

__exclude_allocated_pci_device_from_shared_nic did:

delegated_capacity -= allocated.get_capacity_allocations()

Capacities defines __sub__ but not __isub__, so this builds a new object and rebinds the local — the delegation keeps its original unit count, while the BDF list beside it is mutated in place.

Consequence: a Shared NIC with every BDF allocated was never excluded from the candidate node, because the capacity check still reported the original units. The next allocation then reached usable_bdfs[0] on an empty list and raised IndexError instead of a clean INSUFFICIENT_RESOURCES. The comment at the capacity check — "Delegated capacity would have been decremented already to exclude allocated shared NICs" — shows the decrement was meant to persist.

The remaining BDFs are the source of truth, so the fix derives the unit count from them and mutates the delegated object in place, exactly the way the BDF list is already updated. Also guards a missing or empty label delegation, which previously raised AttributeError.

This bug predates the mixed-mode work and is independent of it.

Tests

New fabric_cf/actor/test/unit/test_mixed_mode_nic.py, 9 cases, no infrastructure required.

Mixed-mode:

  • the dedicated half allocates its single port, MAC and local_name
  • a two-port ConnectX-6 request does not match the single-port ConnectX-6-1P card
  • allocating either half leaves the other untouched
  • exhausting every VF of PF0 removes only the shared half; PF1 keeps its BDF and unit

Capacity tracking:

  • delegated capacity follows the remaining BDFs at every step
  • the last BDF reports exhausted, earlier ones do not
  • an unknown BDF leaves capacity alone
  • an exhausted card raises INSUFFICIENT_RESOURCES rather than IndexError

4 of these fail against the old code and pass against this branch. The full fabric_cf/actor/test/unit/ directory is green (39 passed).

Version

Bumped to 2.0.1b1 for dev deployment. 2.0.1 is not yet on PyPI, so 2.0.1b1 sorts ahead of the released 2.0.0 and behind the eventual 2.0.1. Local image tags in the three docker-compose.yml files follow the version.

⚠️ Two commits to revert before merge

  • DEV ONLY: pin fabric-fim to 2.0.1b1
  • DEV ONLY: pin fabric-am-handlers to 2.0.2b1 in Dockerfile-auth

Both exist so the branches can be deployed together on dev. main should not depend on or build against pre-releases — drop both before merging.

Depends on

fabric-testbed/InformationModel#223

🤖 Generated with Claude Code

https://claude.ai/code/session_016wB1CV7hRtwmWVLPY8GBxh

kthare10 and others added 5 commits September 10, 2026 22:42
A ConnectX-6 with PF0 in SR-IOV mode and PF1 dedicated is advertised as two
independent components. No broker change was needed: model matching is by
string, SmartNIC port matching is by local_name and so port-count agnostic,
and components are excluded by name. These tests pin that behaviour down.

Covers: the dedicated half allocating its single port; a two-port
ConnectX-6 request not matching the single-port ConnectX-6-1P card;
allocating either half leaving the other untouched; and exhausting every VF
of PF0 having no effect on PF1.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wB1CV7hRtwmWVLPY8GBxh
__exclude_allocated_pci_device_from_shared_nic decremented the delegated
capacity with `delegated_capacity -= allocated.get_capacity_allocations()`.
Capacities defines __sub__ but not __isub__, so that built a new object and
rebound the local while the delegation kept its original unit count - even
though the BDF list beside it was being mutated in place.

A Shared NIC with every BDF allocated was therefore never excluded from the
candidate node, because the capacity check still reported the original
units. The next allocation reached usable_bdfs[0] on an empty list and
raised IndexError instead of INSUFFICIENT_RESOURCES. The comment at the
capacity check - "Delegated capacity would have been decremented already to
exclude allocated shared NICs" - shows the decrement was meant to persist.

The remaining BDFs are the source of truth, so derive the unit count from
them and mutate the delegated object in place, the same way the BDF list is
already updated. Also guards a missing or empty label delegation, which
previously raised AttributeError.

Four of the tests in test_mixed_mode_nic.py fail against the old code and
pass against this.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wB1CV7hRtwmWVLPY8GBxh
Lets the mixed-mode NIC branches be deployed together on dev. REVERT THIS
COMMIT before merge - main should not depend on a pre-release.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wB1CV7hRtwmWVLPY8GBxh
Pre-release so the mixed-mode NIC branches can be deployed and tested on dev
together. 2.0.1 is not yet on PyPI, so 2.0.1b1 sorts ahead of the released
2.0.0 and behind the eventual 2.0.1. Local image tags follow the version.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wB1CV7hRtwmWVLPY8GBxh
Matches the AMHandlers pre-release carrying the PCI rescan scoping fix.
fabric-am-handlers 2.0.1 is already on PyPI, so the pre-release has to be
2.0.2b1 rather than 2.0.1b1 to sort after it.

REVERT THIS COMMIT before merge - main should not build against a
pre-release.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wB1CV7hRtwmWVLPY8GBxh
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant