Skip to content

[azure-ai-projects] Emit SDK and add samples and tests - #48774

Draft
Glenn Harper (glharper) wants to merge 7 commits into
feature/azure-ai-projects/vnextfrom
glharper/emit-from-typespec-26-08-1619
Draft

[azure-ai-projects] Emit SDK and add samples and tests#48774
Glenn Harper (glharper) wants to merge 7 commits into
feature/azure-ai-projects/vnextfrom
glharper/emit-from-typespec-26-08-1619

Conversation

@glharper

Copy link
Copy Markdown
Member

Summary

  • emit azure-ai-projects from Azure/azure-rest-api-specs commit 592d36dbd539e1365d0e8c0e993daf8ffeda29ef
  • apply package post-emitter fixes, preview feature-header wiring, and regenerate APIView stubs
  • add synchronous and asynchronous WebIQ and shell-toolbox samples
  • add complete recorded service tests that remain skipped until recordings are added
  • update the 2.6.0 changelog

Skills used

  • azure-ai-projects-emit-from-typespec
  • azure-ai-projects-author-samples
  • azure-ai-projects-author-tests
  • azure-ai-projects-update-changelog

This branch was based on the authoring-skills branch, so it also contains the skill definitions submitted separately in #48772.

Validation

  • 190 preview feature-header tests passed
  • 4 new recorded service tests collect and skip as intended
  • pylint passed
  • source and newly added sample mypy checks passed

The full sample mypy run still reports pre-existing OpenAI union-typing errors in three multi-agent samples; none involve the new files.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 5128bf4a-b1c4-4e61-8806-c895a3362cfb
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 5128bf4a-b1c4-4e61-8806-c895a3362cfb
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 5128bf4a-b1c4-4e61-8806-c895a3362cfb
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 5128bf4a-b1c4-4e61-8806-c895a3362cfb
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 5128bf4a-b1c4-4e61-8806-c895a3362cfb
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 5128bf4a-b1c4-4e61-8806-c895a3362cfb
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 5128bf4a-b1c4-4e61-8806-c895a3362cfb
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
10 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@github-actions

Copy link
Copy Markdown
Contributor
[Pilot] PR Pipeline Failure Analysis

What failed

Pipeline: python - pullrequest (Build ID 6750191)

All three test platforms failed with the same root cause — playback failure due to a <Foundry-Features> header mismatch between recorded sessions and the current SDK:

Platform Failed tests (approx.)
macos311 (whl + sdist) ~136
Ubuntu2404_313 (whl + sdist) ~136
windows2022_312 (whl + sdist) ~136

Three failure groups, one root cause:

  1. Telemetry tests (test_ai_agents_instrumentor, test_responses_instrumentor, test_responses_instrumentor_workflow, and their _async variants) — 16 tests × platforms
  2. Agent CRUD tests (test_agent_disable_enable, test_prompt_agent_endpoint_responses, test_prompt_agent_endpoint_responses_async) — 3 tests × platforms
  3. Sample tests (test_agent_tools_samples, test_agents_samples, test_hosted_agents_samples, and their _async variants) — all sample tests × platforms
Relevant pipeline output
azure.core.exceptions.ResourceNotFoundError: Playback failure -- for help resolving,
see https://aka.ms/azsdk/python/test-proxy/troubleshoot. Error details:
Unable to find a record for the request POST
(sanitized.services.ai.azure.com/redacted)
test-workflow-agent/versions?api-version=v1
Header differences:
    <Foundry-Features> values differ,
    request <WorkflowAgents=V1Preview,ExternalAgents=V1Preview,DraftAgents=V1Preview,
              AgentsOptimization=V2Preview,ModelRouterControls=V1Preview>,
    record  <WorkflowAgents=V1Preview,ExternalAgents=V1Preview,DraftAgents=V1Preview,
              AgentsOptimization=V2Preview>
Body differences:
Remaining Entries:
0: (sanitized/redacted)
1: (sanitized/redacted)

Root cause

The new TypeSpec emit added ModelRouterControls=V1Preview to the <Foundry-Features> request header. Every existing test recording was captured without this feature flag, so test-proxy cannot match the incoming requests to the stored cassettes and returns 404. This causes all 136+ playback-mode tests to fail identically across all three platforms and both install formats (whl, sdist).

Recommended next steps

  • Re-record all affected test sessions with live Azure credentials so recordings include the new ModelRouterControls=V1Preview feature flag. Run tests in record mode:
    cd sdk/ai/azure-ai-projects
    AZURE_TEST_RUN_LIVE=true pytest tests/agents/telemetry/ tests/agents/test_agents_crud.py tests/samples/ --record-mode=record
  • Commit the updated recording files in sdk/ai/azure-ai-projects/tests/recordings/.
  • Alternatively, if ModelRouterControls=V1Preview should be excluded from header matching in the test proxy, add a sanitizer/matcher configuration to strip or ignore this header during playback.
  • See https://aka.ms/ci-fix

Copilot detected the failing pipeline and generated the analysis above. To have it attempt a fix automatically, reply with @copilot please fix the failing pipeline on this PR.

Generated by Pipeline Analysis Next Steps · 70.3 AIC · ⌖ 9.07 AIC · ⊞ 6K ·

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant