chore: optimize CI test coverage

[cloudsmith-iduffy]

Summary

Reduce PR CI cost and feedback time without reducing the Python compatibility guarantee or the binary targets validated before release.

• keep hermetic pytest coverage across Python 3.10-3.14 on every PR
• run live Cloudsmith integration tests once on Python 3.12 instead of once per interpreter
• build representative Linux, macOS, and Windows binaries on PRs
• retain the complete seven-target binary matrix on master, reusable release runs, and manual runs
• calculate coverage once on Python 3.12 instead of five times
• remove real exponential-backoff sleeps from tests while preserving retry assertions
• add stale-run cancellation and Docker layer caching to ancillary workflows

Motivation

The existing matrix provided good coverage, but several dimensions repeated work without testing a distinct compatibility property:

1. The live integration cases exercise the Cloudsmith service and CLI behavior, not interpreter-specific behavior. Running them concurrently on all five Python versions increased external API traffic and resource mutation without materially increasing confidence.
2. Standalone binaries embed Python 3.12. Running all architecture/libc combinations remains important, but doing all seven builds for every source PR is expensive. Three representative PR targets preserve Linux, macOS, and Windows feedback; the full target matrix still runs after merge and for the exact release artifacts.
3. Coverage instrumentation is interpreter-independent for this suite, so collecting the same report five times only consumes CPU.
4. Three HTTP-error tests were waiting through production retry backoff. Together they accounted for roughly 21 seconds of a ~36-second serial test run even though the tests only needed to verify retry count and error handling.

Why keep all supported Python versions on PRs?

The package declares support for Python 3.10 through 3.14. Dependency resolution, SDK behavior, standard-library differences, and syntax/runtime compatibility can vary across those versions. The hermetic suite is now fast enough that retaining all five interpreters gives useful early confidence at relatively low cost.

This deliberately reduces platform breadth for PR binary builds, not Python compatibility breadth.

Test organization

The suite now has an explicit integration marker:

• 856 hermetic tests run across Python 3.10-3.14
• 31 tests that call the live Cloudsmith service run once on Python 3.12
• fork PRs skip integration setup when credentials are unavailable
• ten mocked unit cases no longer depend on live credential fixtures

Environment fixtures now restore their changes through monkeypatch, and pytest collection is scoped to cloudsmith_cli so local tool worktrees cannot be collected accidentally.

Measured impact

• hermetic suite without coverage: 856 passed in 8.84s, down from approximately 36s
• hermetic suite with coverage: 856 passed in 15.55s, 75% total coverage
• integration boundary: 31 explicitly collected live cases
• retry-focused tests: 3 passed in 0.42s

Tradeoff

A binary regression specific to Linux ARM, musl, or macOS Intel may now first appear in the full master matrix rather than in the originating PR. This is intentional: representative OS coverage remains on PRs, while complete target validation remains mandatory on the default branch and release workflow.

Validation

• pytest -m 'not integration' --no-cov
• pytest -m 'not integration' --cov=cloudsmith_cli
• integration marker collection verified separately
• pre-commit run --all-files

[Copilot]

Pull request overview

Optimizes CI runtime by separating hermetic and live tests, reducing redundant coverage and PR binary builds, and adding workflow caching and cancellation.

Changes:

• Adds integration markers and safer, faster test fixtures.
• Runs coverage once and limits PR binaries to representative targets.
• Preserves full post-merge validation and adds workflow optimizations.

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Summary
pyproject.toml	Configures pytest markers and scoped discovery.
cloudsmith_cli/core/tests/test_rest.py	Removes retry backoff delays.
cloudsmith_cli/core/tests/test_download.py	Removes unused temporary-directory setup.
cloudsmith_cli/cli/tests/conftest.py	Restores environment changes safely.
cloudsmith_cli/cli/tests/commands/test_upstream.py	Marks live tests as integration tests.
cloudsmith_cli/cli/tests/commands/test_tokens.py	Isolates mocked API behavior.
cloudsmith_cli/cli/tests/commands/test_repos.py	Marks live tests as integration tests.
cloudsmith_cli/cli/tests/commands/test_package_commands.py	Marks live tests as integration tests.
cloudsmith_cli/cli/tests/commands/test_login.py	Marks live tests as integration tests.
cloudsmith_cli/cli/tests/commands/test_entitlements.py	Marks live tests as integration tests.
cloudsmith_cli/cli/tests/commands/test_credential_helper.py	Removes retry sleeps from tests.
cloudsmith_cli/cli/tests/commands/test_check.py	Separates mocked and live checks.
cloudsmith_cli/cli/tests/commands/policy/test_vulnerability.py	Marks live policy tests.
cloudsmith_cli/cli/tests/commands/policy/test_licence.py	Marks live policy tests.
cloudsmith_cli/cli/tests/commands/policy/test_deny.py	Marks live policy tests.
cloudsmith_cli/cli/tests/commands/conftest.py	Adds fake API-host fixture support.
AGENTS.md	Documents revised test commands; coverage should retain the integration exclusion.
.github/workflows/ty-check.yml	Adds push runs and stale-run cancellation.
.github/workflows/test.yml	Splits unit, coverage, and integration jobs.
.github/workflows/image-build.yml	Adds BuildKit caching and cancellation.
.github/workflows/binaries.yml	Filters PR targets while retaining full matrices elsewhere.
.github/scripts/binaries-matrix.json	Defines representative PR binary targets.

Suppressed comments (1)

.github/workflows/test.yml:75

• This gate does not check PYTEST_CLOUDSMITH_USERNAME or PYTEST_CLOUDSMITH_PASSWORD, even though both login integration tests require those fixtures. If the API key, host, and organization are configured but either login credential is missing, the integration job runs green with those tests skipped, so the full live suite is not actually validated. Include these two secrets in the gate condition as well.

          if [ -n "${API_KEY}" ] && [ -n "${API_HOST}" ] && [ -n "${ORGANIZATION}" ]; then

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.