feat: allow SSH clones from custom git servers via GIT_CLONE_EXTRA_HOSTS

[gxc]

Why: SSH paths are needed

generate_cpg only ever accepted https://github.com/... and https://gitlab.com/… — a hardcoded two-host allowlist. That locks the server out of the deployments where it is arguably most useful:

• Self-hosted git servers cannot be analyzed at all. A Forgejo / Gitea / GitLab instance on the LAN is unreachable by construction, even when the operator explicitly wants it analyzed. There is no way to opt in a new host today.
• SSH is the natural (often the only sane) auth path for self-hosted instances. They typically expose git over ssh:// on a custom port with key-based auth. Forcing HTTPS would mean minting and rotating HTTP tokens on the git server and feeding them through the MCP; an operator deploy key needs none of that — and the credential never appears in a URL, an argument, or .git/config.
• Proprietary code can stay on the operator's network. Internal repos can be analyzed without the source (or a PAT) ever transiting a third party.
• ssh://git@host:3000/owner/repo.git carries custom ports unambiguously — the common shape for a self-hosted instance — unlike scp-style git@host:path, which stays rejected.

This PR therefore extends the clone allowlist in an operator-configured, opt-in way via GIT_CLONE_EXTRA_HOSTS. With the variable unset, behavior is unchanged from main.

What changed (vs. main)

Validation — src/utils/validators.py

• validate_repo_url() restructured into explicit scheme branches:
  • github.com / gitlab.com keep main's exact posture: https-only, default port only, no embedded credentials, canonical lowercase https://<host>/ literal prefix gate.
  • ssh:// is accepted only for hosts listed in GIT_CLONE_EXTRA_HOSTS: a username is allowed, an embedded password is rejected, the hostname must match an entry exactly (look-alike / smuggling hosts still rejected), and the path must still be at least /owner/repo. http(s) to a custom host is rejected — extra hosts are ssh-only.
• New GIT_CLONE_EXTRA_HOSTS parser (_extra_repo_host_entries() / is_extra_repo_host()): comma-separated host / host:port entries, IPv6 in brackets. A bare host allows any port on it; host:port pins that port. Malformed entries raise ValidationError so a typo'd config fails loudly instead of silently widening the allowlist.

Cloning — src/services/git_manager.py

• New _ssh_clone_env() builds the GIT_SSH_COMMAND used for the clone: the full GIT_CLONE_SSH_COMMAND override wins, otherwise it is assembled from GIT_CLONE_SSH_KEY_PATH plus -o BatchMode=yes (a missing key fails fast instead of hanging on a prompt) and -o StrictHostKeyChecking=accept-new (host key recorded on first contact). The env is passed to git.Repo.clone_from().
• Token injection is now scoped to http(s) URLs and the token is URL-quoted; the .git/config credential strip runs only when a credential was actually injected; ssh:// URLs are passed through untouched (auth rides in GIT_SSH_COMMAND, never the URL).

Config & deployment

• src/defaults.py: GIT_CLONE_EXTRA_HOSTS, GIT_CLONE_SSH_KEY_PATH, GIT_CLONE_SSH_COMMAND — all default empty = no behavior change.
• docker-compose.yml: mounts ${GIT_CLONE_SSH_KEYS_HOST_DIR:-./.ssh-keys} read-only at /keys in the codebadger-mcp container and derives GIT_CLONE_SSH_KEY_PATH=/keys/id_ed25519 in-container. GIT_CLONE_SSH_KEY_PATH is deliberately not passed through — a host path never resolves inside the container.
• Dockerfile.mcp: adds openssh-client. .gitignore: ignores .ssh-keys/. .env.example: documented, commented examples for both host-run and dockerized MCP.
• src/tools/core_tools.py: generate_cpg docstring / field descriptions updated so MCP callers learn that ssh:// URLs on configured hosts are accepted.

Docs

• docs/deployment.md: new "Custom git servers (GIT_CLONE_*)" section with a variable table and the host-run-MCP vs. dockerized key handling explained.
• docs/security.md: control ①a updated — the SSRF posture is unchanged; the extension is explicit, operator-scoped, exact-host, and ssh-only.

Tests

• New tests/test_git_manager.py: GIT_SSH_COMMAND construction and override precedence, ssh:// URL passthrough with no credential stripping, historic github-token injection unchanged, custom-host http(s) rejected before any clone.
• tests/test_validators.py (TestExtraGitHosts): accept/reject matrix for ssh/http(s) across host/port/IPv6 entries, port pinning, ssh:// still rejected for github.com/gitlab.com, malformed configs fail loudly, and the empty-config default posture is unchanged.

Testing

• Full unit suite (pytest tests/ --ignore=tests/integration): 630 passed, 35 skipped, 0 failed — the skips are environment-gated (Postgres/Redis/live-Joern tests). Includes the 17 new tests covering the accept/reject matrix and the clone env behavior, plus test_compose_security.py over the compose changes. The live-server integration suite is exercised in CI.

Security posture

Default allowlist unchanged (github.com/gitlab.com, https-only). The extension is opt-in, operator-scoped, exact-host, ssh-only, with no credentials ever placed in URLs; control characters, look-alike domains, userinfo smuggling, and short paths are still rejected.

[Lekssays]

Reviewed this closely and it's solid work — thanks. The default posture is provably unchanged: I differential-tested validate_repo_url on main vs. this branch with an empty config across 41 URLs (homoglyphs, userinfo smuggling, ports, control chars, scp-style, file://, metadata IPs, case variants) and the output is byte-identical. The bypass probe with hosts configured also came back clean — look-alikes, forge.lan@evil.com, trailing dot, percent-encoded host, embedded password, http(s)-to-custom-host all rejected.

Also worth calling out: quote(token, safe='') is a real latent bug fix — a PAT containing /, @, # or ? previously corrupted the clone URL. And gating the .git/config strip on injected_credential rather than token is the right refactor.

I've pushed a follow-up commit to this branch with the review items rather than sending you round-trips. Summary:

Ports are now part of the allowlist. This was the one thing I'd call a genuine widening: a bare forge.lan entry permitted ssh://git@forge.lan:<any of 1–65535>/o/r. Since source_path is caller-influenceable in a chat-facing MCP, that turned one allowlisted host into an ssh-speaking port scanner. A bare entry now means port 22 only; host:port pins as before; new host:* opts into any port explicitly.

Falling out of that:

• Port 0 is rejected (urlparse only range-checks the upper bound, so :0 slipped past the entry parser's own 1..65535 check).
• A portless ssh URL now matches against 22 — what a clone actually dials — so a natural forge.lan:22 entry no longer rejects ssh://git@forge.lan/owner/repo. That trap was baked into test_port_pinning_enforced.

Config validation moved to startup. _extra_repo_host_entries() only ran on the ssh path, so a typo'd GIT_CLONE_EXTRA_HOSTS left github clones working and surfaced only on the first custom-host clone — and echoed the raw config entry back to an untrusted caller. It's now parsed in app_lifespan (fails the boot), and the request-time message is redacted with the detail logged for the operator.

_ssh_clone_env returns only GIT_SSH_COMMAND. GitPython layers env over os.environ ("If some variable is not specified in env and is defined in os.environ, value from os.environ will be used"), so copying the whole environment — POSTGRES_PASSWORD, GITHUB_TOKEN, JOERN_SERVER_AUTH_PASSWORD — into the Git object was needless secret spread. Key path is shlex.quoted too, since git runs the command through a shell.

New GIT_CLONE_SSH_KNOWN_HOSTS pins the server key (StrictHostKeyChecking=yes). Unset keeps accept-new, but the docs now say plainly that in the dockerized stack that record lives in the container and dies on every recreate — so it's TOFU per deploy, not first-contact pinning.

Smaller ones: ssh usernames must start alphanumeric (a leading - reaches ssh's argv as an option — git blocks it downstream, but this matches the file's belt-and-suspenders posture); IPv6 hosts normalize through ipaddress so [::1] and [0:0:0:0:0:0:0:1] are one entry; .ssh-keys/ is tracked via .gitkeep so compose doesn't create the default bind mount root-owned in the working tree; docs/security.md ①a now scopes the "two independent gates" claim to the built-in https hosts (the ssh path is the hostname gate plus the port).

Docs (.env.example, deployment.md, security.md, defaults.py) are updated to the new port semantics throughout.

Tests: 634 passed, 35 skipped (up from 628 — six new cases covering the port surface, port 0, option-shaped usernames, IPv6 equivalence, startup validation, and error redaction). Two test_mcp_tools.py failures reproduce identically on main and are environmental — my checkout path contains playground/. Please sanity-check the new host:* syntax against your Forgejo setup; if your entry was a bare host relying on a non-22 port, it now needs the port spelled out.