Skip to content

Make ow ssh produce a pod you can log into, and not leak it if it can't - #79

Open
vohonen wants to merge 1 commit into
longtermrisk:mainfrom
vohonen:fix/ssh-public-key-and-terminate-on-timeout
Open

Make ow ssh produce a pod you can log into, and not leak it if it can't#79
vohonen wants to merge 1 commit into
longtermrisk:mainfrom
vohonen:fix/ssh-public-key-and-terminate-on-timeout

Conversation

@vohonen

@vohonen vohonen commented Aug 18, 2026

Copy link
Copy Markdown

Two related failures. On a RunPod account you don't administer, ow ssh gives
you a machine you cannot reach, and then leaves it running.

Authorise the caller's key:

  • ow ssh never passed PUBLIC_KEY to create_pod, so it relied entirely on
    the key being registered in the RunPod account settings. On a shared account
    that assumption fails and you get a pod nobody can log into.
  • entrypoint.sh already honours PUBLIC_KEY and writes it to
    authorized_keys, so plumbing it through is enough. It travels as an explicit
    public_key= argument rather than inside env, because start_worker
    rebuilds env from os.environ in dev mode and would drop it.
  • New --pubkey, defaulting to <key-path>.pub. If no public key is found we
    warn and carry on rather than failing, since relying on account-registered
    keys is the only thing that works for callers without a local public key, and
    entrypoint.sh overwrites authorized_keys rather than appending to it.

Don't leave a pod billing. There were three ways to lose one:

  • wait_for_ssh had no try/except around it, so a timeout propagated out
    before terminate().
  • bootstrap_remote called sys.exit() on any failing step, which skipped the
    caller's terminate entirely. This one is the nastiest, because the machine is
    reachable and healthy, so nothing looks wrong. A missing unison in the image
    is enough to trigger it, in every mode rather than just --sync. It now raises
    RemoteBootstrapError and lets the caller decide.
  • Both paths now go through one _terminate_after_failure helper, which is a
    no-op for --existing connections, since a machine we did not create is not
    ours to terminate.

wait_for_ssh also discarded stderr, so a rejected key was indistinguishable
from a slow boot: BatchMode=yes fails instantly and the loop just retried for
the full 180 s. It now keeps the last SSH error, reports it, and says outright
when the failure looks like authentication rather than boot time.

Verified against live RunPod pods, both halves:

  • provisioning connects on the first attempt, where it previously could not
    authenticate at all;
  • pointing --pubkey at a key we do not hold the private half of reports
    Permission denied (publickey,password) and says it is authentication rather
    than a slow boot, then terminates the pod on the spot.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com

…an't

Two related failures. On a RunPod account you don't administer, `ow ssh` gives
you a machine you cannot reach, and then leaves it running.

Authorise the caller's key:

- `ow ssh` never passed `PUBLIC_KEY` to `create_pod`, so it relied entirely on
  the key being registered in the RunPod account settings. On a shared account
  that assumption fails and you get a pod nobody can log into.
- `entrypoint.sh` already honours `PUBLIC_KEY` and writes it to
  `authorized_keys`, so plumbing it through is enough. It travels as an explicit
  `public_key=` argument rather than inside `env`, because `start_worker`
  rebuilds `env` from `os.environ` in dev mode and would drop it.
- New `--pubkey`, defaulting to `<key-path>.pub`. If no public key is found we
  warn and carry on rather than failing, since relying on account-registered
  keys is the only thing that works for callers without a local public key, and
  `entrypoint.sh` overwrites `authorized_keys` rather than appending to it.

Don't leave a pod billing. There were three ways to lose one:

- `wait_for_ssh` had no `try`/`except` around it, so a timeout propagated out
  before `terminate()`.
- `bootstrap_remote` called `sys.exit()` on any failing step, which skipped the
  caller's terminate entirely. This one is the nastiest, because the machine is
  reachable and healthy, so nothing looks wrong. A missing `unison` in the image
  is enough to trigger it, in every mode rather than just `--sync`. It now raises
  `RemoteBootstrapError` and lets the caller decide.
- Both paths now go through one `_terminate_after_failure` helper, which is a
  no-op for `--existing` connections, since a machine we did not create is not
  ours to terminate.

`wait_for_ssh` also discarded stderr, so a rejected key was indistinguishable
from a slow boot: `BatchMode=yes` fails instantly and the loop just retried for
the full 180 s. It now keeps the last SSH error, reports it, and says outright
when the failure looks like authentication rather than boot time.

Verified against live RunPod pods, both halves:

- provisioning connects on the first attempt, where it previously could not
  authenticate at all;
- pointing `--pubkey` at a key we do not hold the private half of reports
  `Permission denied (publickey,password)` and says it is authentication rather
  than a slow boot, then terminates the pod on the spot.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant