feat: dockerized migration pipeline with per-project payloads

[hllvc]

Summary

Turns the migrator from a hand-driven terraform apply plus manual editing into an end-to-end, dockerized pipeline: extract from TFC/TFE, enrich with Variable Sets, convert HCL, validate against a schema, and bulk-import into StackGuardian with one command. The transformer itself gains per-project output, a migration summary, API-based state export, VCS trigger mapping, and per-workspace overrides.

Motivation & Context

The previous flow produced a single sg-payload.json keyed by raw TFC project IDs, required the user to pre-create workflow groups and look up their IDs, exported state by running terraform init per workspace (slow, and prone to the plugin-cache concurrency bug), silently dropped information the user needed to act on (sensitive variables, unpinned Terraform versions), and depended on the host having the right Terraform, jq, hcl2json and sg-cli versions. This PR removes those manual steps and makes the run reproducible across Linux, macOS and Windows.

Changes Made

Transformer (transformer/terraform-cloud/)

• Emit one sg-payload.<project>.json per TFC project via a new tfe_projects data source; WorkflowGroup.name is now the human-readable tfc-<project> instead of the project ID.
• Add migration-summary.md / .json (rendered from summary.tmpl, replacing workspace.tmpl) listing skipped sensitive variables, Terraform-version fallbacks, renamed workspaces, and state-export failures.
• Rewrite state export as a null_resource that pulls each workspace's state directly from the TFC API (current-state-version -> hosted download URL) with curl/jq. Idempotent by workspace name/id, forceStateRefresh re-pulls, and per-workspace failures go to state-export-failures.log instead of aborting the apply. The token is read at runtime from the terraform login creds file or TFE_TOKEN, so it never lands in TF state.
• Map TFC VCS settings (tracked branch, push, PR speculative plans, file trigger patterns, auto-apply) to a VCSTriggers block per workflow, gated by SGDefaultEnableVCSTriggers.
• Skip sensitive terraform/env variables (TFC never returns their values) and record them in the summary.
• Map pinned semver terraform_version to TERRAFORM-<version>; anything else (latest, constraints) falls back to the new SGDefaultTerraformVersion.
• Sanitize ResourceName to SG's charset and 100-char limit with deterministic collision suffixes; any actual rename is reported.
• New inputs: tfHostname, SGDefaultTerraformVersion, SGDefaultEnableVCSTriggers, forceStateRefresh, and a typed workspaceOverrides map (integrations, approvers, runners, version, extra env vars, triggers) that takes precedence over SGDefault* per workspace. Requires Terraform >= 1.3 for optional().
• Update example_payload.jsonc and terraform.tfvars.example for the new shape and variables.

Orchestration and tooling

• sg-migrate.sh (host entrypoint) runs scripts/migrate.sh inside the new Dockerfile image, bind-mounting the repo and forwarding SG_API_TOKEN, SG_ORG, SG_BASE_URL, TFE_TOKEN and TF_TOKEN_*. Falls back to native when --native/--local, SG_NATIVE=1, clean, or no Docker.
• scripts/migrate.sh implements init|apply|enrich|convert|validate|import|triggers|all|clean. Import resolves each project's tfc-<project> group, shows an exists/create plan, prompts (skip with -y), creates missing groups via the SG API, imports with sg-cli under retry/backoff, then registers VCSTriggers in a second pass through the webhooks endpoint because bulk create drops them (triggers runs that pass on its own, --no-vcs-triggers skips it). Convert and import run in parallel; .sg/workflow-groups.json (or --mapping) overrides target groups per project, and --export-dir relocates the output directory.
• scripts/tools.sh provides PATH-or-cache tool resolution with pinned downloads of jq, hcl2json, yajsv and the Go sg-cli into .sg/cached/, plus retry and colored logging helpers. The image bundles the same tools (yajsv built from source for arm64).
• scripts/enrich_variable_sets.sh merges TFC Variable Set variables into the payloads via the TFC API, resolving global/project/workspace scope with TFC precedence and reporting sensitive vars and key conflicts.
• Add schema/sg-payload.schema.json (draft-07, derived from the SG OpenAPI Workflow schema) and scripts/validate_payload.sh using yajsv.
• Move convert_hcl_to_json.sh into scripts/ and rework it to run per payload file with an atomic in-place rewrite.
• .gitignore adds .sg/ (tool cache and group map); new .dockerignore keeps .git, .sg, exports, tfvars and Terraform state out of the image context.

Docs

• README.md: new "Quick start (orchestrated)" section, TFE_TOKEN as the recommended auth path, per-file convert/validate/import steps, Go-binary sg-cli instructions, and a "Notes and limitations" section.

Testing

• ./sg-migrate.sh all against a TFC org with multiple projects produces one payload per project, a summary, and state files; import creates the missing tfc-<project> groups and workflows.
• Same run with --native on a host without Docker downloads tools into .sg/cached/ and behaves identically.
• Re-running apply without forceStateRefresh does not re-download state; with it set, all states are re-pulled.
• Workspaces with sensitive variables, latest Terraform version, and Variable Sets at each scope show up correctly in migration-summary.md and the payloads.
• VCS triggers appear on imported workflows and the repo webhook is registered.
• ./scripts/validate_payload.sh export/sg-payload.*.json passes on generated payloads.
• terraform fmt -check on the transformer module.

Risks & Edge Cases

• Output layout changed: a single sg-payload.json became sg-payload.<project>.json per project, and WorkflowGroup.name is tfc-<project> rather than the TFC project ID. Anyone scripting around the old file name needs to adjust.
• The import phase now writes to the SG org (creates workflow groups) before the bulk import. --no-create-groups restores the old requirement that groups pre-exist.
• State export depends on curl/jq being on PATH at apply time; the orchestrator and image guarantee this, a bare terraform apply does not.
• Non-remote execution mode workspaces may have no or stale state in TFC; the summary flags them.
• The JSON schema is intentionally lenient (unknown fields allowed), so validation catches shape errors, not every API constraint.

Deployment Notes

• New env vars: SG_API_TOKEN, SG_ORG, optional SG_BASE_URL (non-prod targets), TFE_TOKEN (recommended over terraform login), and tuning knobs SG_RETRIES, SG_RETRY_BASE, SG_TF_PARALLELISM, SG_CONCURRENCY, SG_NATIVE, SG_IMAGE, SG_EXPORT_DIR, SG_WFGROUP_MAP, SG_CACHE_DIR.
• Docker image is built locally as stackguardian/migrator:local on first run (--build to rebuild); nothing is pushed to a registry.