UN-4009 [MISC] Generate and commit the API deployment OpenAPI spec in-repo

[chandrasekharan-zipstack]

What

The OpenAPI description of the API deployment execute / status endpoints is now generated and committed in this repository, and a test fails when it drifts from the code. drf-yasg, which generated nothing anyone read, is removed.

Why

The published Python client and the SDK generated for it are built from that description. It was produced by a script living outside this repository, so a route, serializer or response change here could silently invalidate it — the breakage would surface later, in a client repo, against a spec nobody in the PR could see.

How

• drf-spectacular added as a backend dependency; DEFAULT_SCHEMA_CLASS and SPECTACULAR_SETTINGS set in settings/base.py. Both are read only during schema generation — no request-time behaviour changes. The schema class is drf_standardized_errors.openapi.AutoSchema, matching the exception handler these views actually run behind.
• api_v2/openapi_schema.py holds the schema annotation. The introspected schema is wrong in ways that matter to a generated client: a bare FileField maps to format: uri (right for output, wrong for a multipart upload), result: null while an execution is pending crashes a generated deserialiser without allow_null, and operation_id / tags decide the command names and module paths clients expose. It is a module of its own rather than part of serializers.py, because none of these serializers parses a request or builds a response.
• api_v2/deployment_spec_urls.py — a urlconf mirroring the real mount. Generating against the included sub-urlconf drops the prefix and produces paths the server does not serve. SPEC_URLCONFS is a tuple: widening the spec to another endpoint is one entry plus its @extend_schema.
• manage.py generate_docstudio_spec writes specs/docstudio-oss.json with sorted keys, so the committed file is a usable drift signal. --check fails instead of writing. Generation refuses to publish on any generator diagnostic, on a spec that is not valid OpenAPI, or on paths carrying a non-default API_DEPLOYMENT_PATH_PREFIX.
• api_v2/tests/test_docstudio_spec.py regenerates and compares, and — because drift alone passes on a spec that is uniformly wrong — anchors the upload encoding, the nullable result, the documented file-result fields and the error body to the code that produces them. It runs in the existing unit-backend group — no database, no new CI job. Its failure message names the repos that regenerate from the spec, since propagating the change downstream is the part CI cannot do.
• drf-yasg and the docs app removed. The redoc UI they served was built with public=False and no endpoint carried a @swagger_auto_schema, so an anonymous caller saw an empty schema. Nothing generated or consumed it. backend/README.md now points at the committed spec and the regeneration command instead of the deleted /api/v1/doc/ route.

Regenerating

uv run python manage.py generate_docstudio_spec

from backend/, then commit. The test says so in its failure message, along with the client and CLI repos that regenerate from the result.

Can this PR break any existing features. If yes, please list possible items. If no, please explain why.

The /doc/ route is gone with drf-yasg, and drf-spectacular serves no UI in its place — the spec is a committed file, not an endpoint. Nothing else changes at request time: the settings added are read only during schema generation, and no route or response is modified. The only serializer change is help_text on the two hitl_* fields, which is published documentation and does not affect validation. The annotation is metadata on the view, not behaviour.

Database Migrations

None.

Env Config

None. API_DEPLOYMENT_PATH_PREFIX is read as before; generation now refuses to write a spec when it is overridden away from deployment, so a developer's local override cannot reach the committed artifact.

Relevant Docs

• backend/README.md — API Docs section, replaced with the committed spec's location and the regeneration command.

Related Issues or PRs

• UN-4009 (sub-task of UN-4008)
• UN-4010 [FEAT] Support every API deployment request parameter via a generated transport unstract-python-client#27 — the client generated from this spec

Dependencies Versions

Package	Before	After
drf-yasg	1.21.15	removed
drf-spectacular	—	0.30.0 (pinned exactly: the generated artifact is committed, so a minor bump would show as spec drift rather than as a dependency change)

drf-standardized-errors is unchanged at >=0.12.6; it was already a dependency and is now also the source of the schema class. Transitively, drf-yasg required inflection, packaging, pytz, pyyaml and uritemplate; drf-spectacular requires inflection, jsonschema, pyyaml and uritemplate. All of them except jsonschema were already in the lock through other packages.

Notes on Testing

676 passed, 1 skipped (plus 340 subtests) in the backend unit tier this lands in, including the 18 spec tests. Verified by hand that the two new generation gates fire: a typo in a hand-written schema fragment fails OpenAPI validation, and API_DEPLOYMENT_PATH_PREFIX=private fails the prefix check.

The generated spec is byte-identical to what the external script produced, apart from the root tags array — clients had nowhere to read group descriptions from, which is the one deliberate addition — and the review fixes in the latest commit (error body shape, nullability, the statuses each operation can actually return).

Screenshots

n/a — no user-facing surface.

Checklist

I have read and understood the Contribution Guidelines.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KjRpEocCvnxGUSkFivgnhk

[greptile-apps]

Greptile Summary

The PR replaces the unused runtime documentation endpoint with a committed OpenAPI contract for API deployment execution and adds generation and drift validation.

• Adds explicit request, response, authentication, and error annotations for deployment execution and status operations.
• Adds a management command and unit tests that regenerate and validate specs/docstudio-oss.json.
• Replaces drf-yasg with pinned drf-spectacular and removes the old docs route and app.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains in the eligible follow-up review scope.

No blocking failure remains.

Important Files Changed

Filename	Overview
backend/api_v2/openapi_schema.py	Defines the client-facing deployment request, response, authentication, status, and error schema annotations.
backend/api_v2/management/commands/generate_docstudio_spec.py	Generates deterministic OpenAPI output and rejects diagnostics, invalid schemas, unsupported prefixes, and drift.
backend/api_v2/tests/test_docstudio_spec.py	Checks committed-spec drift and anchors important generated-client contracts to implementation behavior.
backend/api_v2/deployment_spec_urls.py	Selects the actual served deployment route mount for schema generation.
backend/backend/settings/base.py	Configures drf-spectacular with standardized-error schema generation and deployment-key metadata.
backend/pyproject.toml	Replaces drf-yasg with a pinned drf-spectacular production dependency.
specs/docstudio-oss.json	Commits the generated execute and status API contract consumed by downstream clients.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    Views[Deployment views and serializers] --> Annotations[OpenAPI annotations]
    Routes[Served deployment URL configuration] --> Generator[Schema generation command]
    Annotations --> Generator
    Generator --> Validation[OpenAPI and diagnostic validation]
    Validation --> Spec[specs/docstudio-oss.json]
    Spec --> Clients[Published client and generated SDKs]
    Generator --> DriftTest[Unit-test drift check]
    Spec --> DriftTest

Loading

_{Reviews (4): Last reviewed commit: "Merge branch 'main' into feat/docstudio-..." | Re-trigger Greptile}

[ritwik-g]

Standardized review — verdict: REQUEST CHANGES

Critical: 0 · High: 2 · Medium: 8 · Low: 7 · Lenses run: 17/17

Reviewed against a fixed 17-lens rubric (unstract:standard-review, plugin v0.30.1) at 51c7cd45, diffed from merge-base 6b916eac — 15 files, matching GitHub's count.

The engineering here is careful: the guard against spectacular's guessing, the urlconf that selects the real mount rather than restating it, the failure message naming the downstream repos. Two things to say up front:

• The claim that CI fails on drift is real — I suspected it wasn't and checked. tests/groups.yaml:85-94 collects backend/** by glob, and this PR's own CI run shows unit-backend at 1009 passed.
• A dedicated security pass (the plugin's vendored prompt, separate from the lenses) found no findings at confidence ≥ 8 — an empty result within its scope, not by exclusion. /doc/ was never in WHITELISTED_PATHS, the committed spec has no servers block and no secrets, and removing the redoc UI net-reduces attack surface.

Both High findings were confirmed by running the real serializers against the real DTOs in a scratch venv, not by reading. Both reach a generated SDK, and this spec is the artifact Zipstack/unstract-python-client#27 is generated from — so fixing them here will move that PR's generated code.

Unanchored findings

These have no line in this diff to attach to.

• [Medium] [Lens 17] backend/README.md:178-179 now points at a route this PR deletes. "access the API documentation that's auto generated at the backend endpoint /api/v1/doc/" — that route goes with drf-yasg and the docs app (public_urls.py:31, public_urls_v2.py:28). The section now describes an endpoint that 404s, on the page a new backend developer reads first. The only surviving doc/ route is admin/doc/ (Django admindocs), a different mount behind ADMIN_ENABLED. Suggest replacing the section with the committed spec and the regeneration command. (Verified directly — one automated sweep reported no surviving references and was wrong.)
• PR title — this repo states no title convention in writing (no CLAUDE.md, no CONTRIBUTING title section, template links out only), so there is nothing to judge against. MISC is the right type regardless: nothing here changes what the product does for a user.
• PR body drops template sections, including Dependencies Versions — on the one PR that swaps drf-yasg for a pinned drf-spectacular==0.30.0, that's the section this change most needed.
• One description claim is unfalsifiable from this repo: "byte-identical to what the external script produced, apart from the root tags array." The external script isn't here; unstract-python-client#27 is where that gets checked.

Open questions

1. Is a cloud spec generated from this same command? That decides whether the hitl_* finding wants a flat exclude_fields or a conditional.
2. Was the {status, message} ErrorResponse shape a deliberate simplification, or written before drf-standardized-errors became the handler?

Assumptions

• drf-standardized-errors is the active handler for these endpoints (verified at backend/middleware/exception.py:48). If a deployment overrides EXCEPTION_HANDLER, the first finding's severity changes.
• specs/docstudio-oss.json is the OSS artifact only, per its filename and DEFAULT_OUT. If it also feeds cloud, the hitl_* finding becomes High.

Lens checklist (17/17)

1 see findings · 2 see findings · 3 see findings · 4 Clean — dedicated security pass, plus: all published paths match served routes, spec has no servers/secrets, /doc/ was never whitelisted · 5 N/A — no migrations or persisted state · 6 N/A — no concurrency primitives · 7 see findings; consumer half checked separately — endpoint, status codes and wire format are untouched, so unstract-cloud/load_test and unstract-docs are unaffected; the spec's consumers are the unmerged #27 and the CLI · 8 N/A — generation is offline, no external calls added · 9 Clean · 10 Clean · 11 Clean — backend/docs/ fully removed, zero residual drf_yasg references, no docker/chart references, no request-time behaviour change · 12 Clean — the only agent-surface change is @extend_schema(exclude=True) on MCPServerView, documentation-only · 13 see findings · 14 Clean — pinned with a stated reason, canonical package, lockfile consistent, transitive set shrinks · 15 see findings · 16 see findings · 17 see Unanchored

Posted as COMMENT, not REQUEST_CHANGES — the merge decision is yours, not the review's.

[chandrasekharan-zipstack]

Thanks — this was a genuinely useful review; the two High findings were real and both would have shipped into the generated client. Everything is addressed in eddd4b7. Inline replies cover the anchored threads; the unanchored ones and your open questions:

backend/README.md:178-179 — replaced. The API Docs section now points at specs/docstudio-oss.json as the committed contract and gives the regeneration and --check commands, instead of describing a route that 404s.

PR body — restored to the template, including Dependencies Versions: drf-yasg 1.21.15 removed, drf-spectacular==0.30.0 added (pinned exactly, because the generated artifact is committed and a minor bump would surface as spec drift rather than as a dependency change). Also filled in Env Config, Relevant Docs, Screenshots and the break-risk section.

Open question 1 — is a cloud spec generated from this command? No. This is the single artifact, and the PyPI client generated from it serves cloud users, so the hitl_* parameters stay and instead say what they require. Details on that thread.

Open question 2 — was {status, message} deliberate? Neither, really: it was written from the two hand-built bodies in api_deployment_views.py and then generalised to every failure without checking what the handler emits. Now {status, message} is declared only on the branches that actually build it by hand, and everything else carries the handler's shape.

Assumption you flagged — drf-standardized-errors as the active handler is now load-bearing rather than incidental: DEFAULT_SCHEMA_CLASS names its AutoSchema, and a test replays drf_logging_exc_handler over real exceptions and asserts the emitted keys match the published ErrorResponse/ErrorDetail. A deployment that overrode EXCEPTION_HANDLER would fail that test rather than quietly publish a wrong shape.

Also added while in here, both prompted by your findings rather than requested: generation now rejects a spec whose paths carry a non-default API_DEPLOYMENT_PATH_PREFIX, and the error-type enum is named ErrorType rather than spectacular's default TypeEnum, since that name becomes a class in the generated SDK.

One thing deliberately not done: fetch_presigned_file still passes the upstream response's status through verbatim. Normalising it to a single 502 changes what live callers receive, which does not belong in a spec PR — the spec now declares 413/502/504 and stops asserting a single cause for 403/404. Worth a follow-up issue.

Backend unit tier: 676 passed, 1 skipped (340 subtests), 18 of them spec tests. Both new generation gates verified by hand — a "patern" typo in PATH_SEGMENT now fails OpenAPI validation, and API_DEPLOYMENT_PATH_PREFIX=private fails the prefix check.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[github-actions]

Unstract test results

Per-group results

Status	Group	Tier	Passed	Failed	Errors	Skipped	Duration (s)
✅	e2e-api-deployment	e2e	3	0	0	0	20.3
✅	e2e-coowners	e2e	1	0	0	0	1.4
✅	e2e-etl	e2e	1	0	0	0	8.5
✅	e2e-login	e2e	2	0	0	0	1.2
✅	e2e-prompt-studio	e2e	1	0	0	0	4.7
✅	e2e-smoke	e2e	2	0	0	0	1.3
✅	e2e-workflow	e2e	1	0	0	0	16.3
✅	integration-backend	integration	310	0	0	26	45.0
✅	integration-connectors	integration	1	0	0	7	7.7
✅	integration-workers	integration	157	0	0	1	49.9
✅	unit-backend	unit	1158	0	0	1	41.4
✅	unit-connectors	unit	63	0	0	0	9.7
✅	unit-core	unit	33	0	0	0	1.3
✅	unit-platform-service	unit	15	0	0	0	2.6
✅	unit-rig	unit	117	0	0	0	5.4
✅	unit-runner	unit	5	0	0	0	3.0
✅	unit-sdk1	unit	563	0	0	0	28.7
✅	unit-workers	unit	1397	0	0	1	130.7
	TOTAL		3830	0	0	36	379.2

Critical paths

⚠️ Critical paths not yet covered

• workflow-execution-fan-out — Multi-file workflow execution fans out to file-processing workers and rejoins. (declared coverage: no groups declared)

✅ Covered critical paths

• auth-login — covered by e2e-login
• adapter-register-llm — covered by integration-backend
• workflow-author — covered by integration-backend
• co-owner-manage — covered by integration-backend, e2e-coowners
• workflow-create-execute — covered by e2e-workflow
• api-deployment-provision — covered by integration-backend
• api-deployment-auth — covered by integration-backend
• api-deployment-run — covered by e2e-api-deployment
• mcp-server-auth — covered by integration-backend
• mcp-platform-auth — covered by integration-backend
• prompt-studio-author — covered by integration-backend
• prompt-studio-fetch-response — covered by e2e-prompt-studio
• connector-register-test — covered by integration-backend
• pipeline-etl-execute — covered by e2e-etl
• usage-aggregate-read — covered by integration-backend
• usage-token-tracking — covered by e2e-api-deployment
• callback-result-delivery — covered by e2e-api-deployment

[ritwik-g]

FOLLOWUP review — verdict: APPROVE

Prior findings: 17 · RESOLVED 16 · WAIVED 1 (with a documented rationale) · NOT RESOLVED 0
New findings: none. Scope change: NO.

Verified against 879ec5ab from the current merge-base e452f778. The 12 commits since my review are 11 from a main merge plus one fix commit, eddd4b74. Status here is what the code shows, not what the replies claim.

The two blockers, verified against live behaviour rather than against the tests

ErrorResponse is now {type: ErrorType, errors: [ErrorDetail]}, both required, bound to 400/401/403/404 on both operations and 413/429/502/504 on execute. Rather than trust the new test, I replayed the real drf_logging_exc_handler over six real exceptions — APIException, ValidationError, NotAuthenticated, PermissionDenied, NotFound, Throttled — and the emitted keys match the published component exactly, with all three emitted type values inside the published enum. Leaving code free-form was the right call: it varies (error, invalid, not_found, not_authenticated, permission_denied, throttled), so an enum would have been violated.

I also checked the risk this fix could have introduced — that the hand-built bodies got the new shape by mistake. They didn't: 406 → AcknowledgedResponse{status, message}, status 422/500 → StatusResponse{status, message}, execute 422/500 → ExecuteResponse{message} with no status key, each matching its Response(...) call site.

error / status_api are nullable: true in the committed artifact and both removed from required (which is now just execution_id, execution_status). Confirmed in the regenerated spec, not only in the annotation source.

The new invariants are real guards

Five were each broken and observed to fail: removing @extend_schema_field(OpenApiTypes.BINARY), dropping allow_null from result, re-adding the phantom metrics, and moving 429 back onto status all now fail after regeneration. test_the_documented_error_body_is_the_one_the_handler_sends genuinely replays the live handler rather than restating the implementation. Your claim that "dropping either decorator this PR was written for now fails regardless of regeneration" holds as tested.

The patern typo repro from my finding now produces SpecGenerationFailed: … is not valid under any of the given schemas, and API_DEPLOYMENT_PATH_PREFIX=private is rejected by output assertion — a better lever than the override_settings I suggested, since base_urls bakes the prefix in at import.

The one waiver, and two notes

The HITL fields stay in the OSS artifact because one spec serves both OSS and cloud — excluding them would make the enterprise feature unreachable from the official client. That answers the open question I raised and is a sound trade; the 400 is now a documented precondition via help_text published as the client-facing description, rather than a surprise.

Two things carried forward, neither blocking and neither introduced here: deployment_helper.py:654-658 still passes an upstream document-store status verbatim, so a store answering e.g. 410 still surfaces undeclared (body shape is fine — it's an APIException); and middleware/exception.py:38-47 has an SdkError branch emitting a third shape, which I could not demonstrate is reachable on these two operations.

Suite green: 18 in the spec tests, 260 across api_v2 middleware mcp_server, both drift tests passing so the artifact regenerates byte-identically. CI green including e2e. Nice work on backend/README.md — that closes the stale /api/v1/doc/ pointer cleanly.