Skip to content

Configure Agent Server trace sampling and instrumentation defaults - #48753

Open
Ankit Singhal (singankit) wants to merge 12 commits into
mainfrom
users/singankit/agentserver-100-percent-trace-sampling
Open

Configure Agent Server trace sampling and instrumentation defaults#48753
Ankit Singhal (singankit) wants to merge 12 commits into
mainfrom
users/singankit/agentserver-100-percent-trace-sampling

Conversation

@singankit

@singankit Ankit Singhal (singankit) commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Description

Configure Agent Server Core observability defaults to provide complete agent traces without noisy dependency spans:

  • Use a 100% Azure Monitor trace sampling ratio when no OpenTelemetry sampler is explicitly configured.
  • Preserve explicit sampler configuration, including microsoft.rate_limited, by omitting the default sampling ratio.
  • Disable Azure SDK, HTTPX, Requests, urllib, and urllib3 instrumentation by default.
  • Allow customers to re-enable individual instrumentations through configure_observability(instrumentation_options=...).
  • Refresh the API stub metadata and align the package version with the 2.1.1 changelog entry.

Validation

  • Added regression coverage for Azure Monitor sampling precedence and instrumentation option overrides.
  • Added a behavioral test proving Azure SDK tracing is not enabled by default.
  • Added live Application Insights E2E coverage for default-disabled and explicitly enabled Azure SDK spans.
  • Verified all 53 tracing tests pass.
  • Verified the defaults on a deployed Microsoft Foundry hosted agent by querying its Application Insights trace.

Trace comparison

Before

Trace before disabling default Azure SDK and HTTP instrumentation

After

Trace after disabling default Azure SDK and HTTP instrumentation

All SDK Contribution checklist:

General Guidelines and Best Practices

  • Title of the pull request is clear and informative.
  • There are a small number of commits, each of which have an informative message.

Testing Guidelines

  • Pull request includes test coverage for the included changes.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
9 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Sets Azure Monitor trace sampling to 100% for Agent Server.

Changes:

  • Configures sampling_ratio=1.0.
  • Documents sampling behavior and adds tests.
  • Adds a 2.1.1 changelog entry.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File Description
_tracing.py Configures full Azure Monitor sampling.
test_tracing.py Tests sampling arguments.
README.md Documents sampling behavior.
CHANGELOG.md Records the change.

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Comment thread sdk/agentserver/azure-ai-agentserver-core/azure/ai/agentserver/core/_tracing.py Outdated
Copilot AI review requested due to automatic review settings August 26, 2026 00:07

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

@github-actions

This comment has been minimized.

Copilot AI review requested due to automatic review settings August 26, 2026 05:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Comment thread sdk/agentserver/azure-ai-agentserver-core/api.md Outdated
Comment thread sdk/agentserver/azure-ai-agentserver-core/CHANGELOG.md
@github-actions

This comment has been minimized.

Copilot AI review requested due to automatic review settings August 26, 2026 16:42

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Suppressed comments (1)

sdk/agentserver/azure-ai-agentserver-core/api.md:11

  • This public API change updates api.md, but api.metadata.yml is unchanged and still contains the hash for the previous API surface. The API metadata generator writes apiMdSha256 from the normalized api.md content (eng/scripts/extract_apiview_metadata.py:38-44), so the committed pair is now inconsistent. Regenerate the API stub/metadata and commit the resulting api.metadata.yml to avoid the API consistency check failing.
            instrumentation_options: Optional[dict[str, dict[str, Any]]] = ...,

Copilot AI review requested due to automatic review settings August 26, 2026 16:48

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Suppressed comments (1)

sdk/agentserver/azure-ai-agentserver-core/api.md:11

  • Regenerate and commit api.metadata.yml with this API snapshot change. Its committed apiMdSha256 still describes the previous api.md, while the repository's API-stub flow always hashes the normalized markdown and writes the metadata alongside it (eng/scripts/extract_apiview_metadata.py:38-44; eng/tools/azure-sdk-tools/azpysdk/apistub.py:235-255). Leaving the old hash makes the API snapshot internally inconsistent and will fail the API consistency check.
            instrumentation_options: Optional[dict[str, dict[str, Any]]] = ...,

@github-actions

Copy link
Copy Markdown
Contributor
[Pilot] PR Pipeline Failure Analysis

What failed

Pipeline: python - pullrequest (Build Analyze)
Build ID: 6748495

The VerifyChangelog step failed for package azure-ai-agentserver-core. The changelog entry for version 2.1.0 is dated 2026-08-24, but the pipeline requires that the date for the version being released is the latest (most recent) date in the file — meaning no other entry in CHANGELOG.md should have a later date.

Relevant pipeline output
Verifying changelog for azure-ai-agentserver-core
Found the following change log entry for version '2.1.0' in [sdk/agentserver/azure-ai-agentserver-core/CHANGELOG.md].
-----
## 2.1.0 (2026-08-24)

### Other Changes

- Added compatibility bounds to runtime and development dependencies so installation cannot silently resolve to incompatible releases.
-----
[debug]Verifying as a release build because the changelog entry has a valid date.
##[error]Invalid date [ 2026-08-24 ]. The date for the changelog being released must be the latest in the file.
##[error]PowerShell exited with code '1'.

Recommended next steps

  • Open sdk/agentserver/azure-ai-agentserver-core/CHANGELOG.md and check whether there is another entry above 2.1.0 with a later date (e.g., an unreleased entry with a future date). If so, either remove that entry's date, change it to Unreleased, or reorder entries so 2.1.0 (2026-08-24) is the most recent dated entry.
  • Alternatively, if 2.1.0 is not yet released, consider replacing the date with Unreleased (e.g., ## 2.1.0 (Unreleased)) and use a release date only when the version is ready to ship.
  • See https://aka.ms/ci-fix

Copilot detected the failing pipeline and generated the analysis above. To have it attempt a fix automatically, reply with @copilot please fix the failing pipeline on this PR.

Generated by Pipeline Analysis Next Steps · 23.9 AIC · ⌖ 8.92 AIC · ⊞ 6K ·

Copilot AI review requested due to automatic review settings August 26, 2026 20:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

sdk/agentserver/azure-ai-agentserver-core/api.md:11

  • Updating api.md without regenerating api.metadata.yml leaves the committed apiMdSha256 stale. The repository’s API stub tooling writes both files together (eng/scripts/extract_apiview_metadata.py:38-44), and the package metadata is unchanged in this PR. Please regenerate the API stub and commit the matching metadata file so API consistency/artifact checks use the new surface.
            instrumentation_options: Optional[dict[str, dict[str, Any]]] = ...,

Copilot AI review requested due to automatic review settings August 26, 2026 20:57

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Suppressed comments (2)

sdk/agentserver/azure-ai-agentserver-core/api.md:11

  • The public API snapshot changed, but api.metadata.yml still contains the previous apiMdSha256. The repository's API-stub workflow generates both files together (doc/dev/tests.md:150), so the API consistency check will reject this stale pair. Please rerun azpysdk apistub . from this package and commit the regenerated metadata.
            instrumentation_options: Optional[dict[str, dict[str, Any]]] = ...,

sdk/agentserver/azure-ai-agentserver-core/CHANGELOG.md:3

  • This new unreleased section targets 2.1.1, but the package's dynamic version source remains VERSION = "2.1.0" in azure/ai/agentserver/core/_version.py:5, so builds still identify as the already released 2.1.0. Unreleased changelog versions are kept aligned with _version.py elsewhere (for example, azure-ai-agentserver-optimization and azure-monitor-opentelemetry); please bump the version source to 2.1.1.
## 2.1.1 (Unreleased)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Copilot AI review requested due to automatic review settings August 26, 2026 22:06

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

sdk/agentserver/azure-ai-agentserver-core/tests/test_tracing.py:461

  • OTEL_TRACES_PER_SECOND is not consumed by the Microsoft distro; the rate-limited sampler reads its value from OTEL_TRACES_SAMPLER_ARG (sdk/monitor/azure-monitor-opentelemetry/azure/monitor/opentelemetry/_utils/configurations.py:188-200). As written, this fixture sets an ignored variable and can mislead future changes about how the rate is configured. Use the standard sampler argument variable instead.
                "OTEL_TRACES_PER_SECOND": "5",

Copilot AI review requested due to automatic review settings August 27, 2026 05:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Comment thread sdk/agentserver/azure-ai-agentserver-core/tests/test_tracing_e2e.py Outdated
Copilot AI review requested due to automatic review settings August 27, 2026 06:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

sdk/agentserver/azure-ai-agentserver-core/tests/test_tracing.py:461

  • OTEL_TRACES_PER_SECOND is not the argument used by the Microsoft rate-limited sampler; it reads OTEL_TRACES_SAMPLER_ARG (azure/monitor/opentelemetry/_utils/configurations.py:188-200). As written, this test silently exercises the default value of 5 rather than the supplied configuration, so it would not catch loss of an explicit rate argument.
                "OTEL_TRACES_PER_SECOND": "5",

sdk/agentserver/azure-ai-agentserver-core/tests/test_tracing_e2e.py:414

  • This one-shot query cannot establish that the dependency was not exported: Application Insights ingestion is asynchronous, and the file explicitly allows up to 300 seconds for individual records to become queryable. The control span may be visible before its dependency, allowing this regression test to pass even when Azure SDK instrumentation is still enabled. Poll over the same ingestion window before asserting absence.
        response = logs_query_client.query_resource(
            appinsights_resource_id,
            dependency_query,
            timespan=timedelta(minutes=30),
        )

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6df612ba-a75a-4a45-8fce-a5ed9f329e2f
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6df612ba-a75a-4a45-8fce-a5ed9f329e2f
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6df612ba-a75a-4a45-8fce-a5ed9f329e2f
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 6df612ba-a75a-4a45-8fce-a5ed9f329e2f
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 6df612ba-a75a-4a45-8fce-a5ed9f329e2f
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 6df612ba-a75a-4a45-8fce-a5ed9f329e2f
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 6df612ba-a75a-4a45-8fce-a5ed9f329e2f
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6df612ba-a75a-4a45-8fce-a5ed9f329e2f
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6df612ba-a75a-4a45-8fce-a5ed9f329e2f
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6df612ba-a75a-4a45-8fce-a5ed9f329e2f
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6df612ba-a75a-4a45-8fce-a5ed9f329e2f
Copilot AI review requested due to automatic review settings August 27, 2026 06:17
@singankit
Ankit Singhal (singankit) force-pushed the users/singankit/agentserver-100-percent-trace-sampling branch from 313c2c9 to fb1097e Compare August 27, 2026 06:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

Suppressed comments (1)

sdk/agentserver/azure-ai-agentserver-core/tests/test_tracing_e2e.py:416

  • This single query can pass before Application Insights has indexed the dependency span. The file already treats ingestion as eventually consistent, and observing the control span does not guarantee that every span in the trace is queryable, so this test may report the instrumentation as disabled when the dependency arrives later. Observe the dependency query for a bounded delay before asserting absence, or rely on the deterministic local test instead of presenting this as a live negative check.
        response = logs_query_client.query_resource(
            appinsights_resource_id,
            dependency_query,
            timespan=timedelta(minutes=30),
        )
        dependency_rows = response.tables[0].rows if response.tables else []
        assert dependency_rows == []

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6df612ba-a75a-4a45-8fce-a5ed9f329e2f
Copilot AI review requested due to automatic review settings August 27, 2026 06:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Hosted Agents sdk/agentserver/*

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants