-
Notifications
You must be signed in to change notification settings - Fork 152
fix(run): bail out of run --wait on a wall clock so system sleep can't hang it
#877
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -668,7 +668,9 @@ def execute( | |
| if renderer.is_pretty(): | ||
| pprint( | ||
| f"[bold red]Error: WebSocket timed out after {timeout}s waiting for server response.[/bold red]\n" | ||
| "[yellow]For long-running workflows, increase the timeout: comfy run --workflow <file> --timeout 300[/yellow]" | ||
| "[yellow]For long-running workflows, increase the timeout: comfy run --workflow <file> --timeout 300[/yellow]\n" | ||
| "[yellow]If the machine slept mid-run, the job may still be running — check " | ||
| "`comfy jobs status <id>`. Keep long local batches awake with `caffeinate`.[/yellow]" | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win 🔎 Supported by static analysis🏁 Script executed: sed -n '650,690p' comfy_cli/command/run/__init__.py
sed -n '155,178p' comfy_cli/error_codes.py
rg -n -i 'caffeinate|macos|darwin|windows|linux|keep.*awake|prevent.*sleep' comfy_cli tests README.md docs 2>/dev/null | head -160Repository: Comfy-Org/comfy-cli Length of output: 19896 🏁 Script executed: #!/bin/bash
set -eu
printf '%s\n' '--- timeout call sites ---'
rg -n -A8 -B8 'renderer\.error\(|code="ws_timeout"|def error\(' comfy_cli/command/run/__init__.py comfy_cli 2>/dev/null | head -180
printf '%s\n' '--- renderer and error registry bindings ---'
rg -n -A35 -B10 'class .*Renderer|def error\(|ERROR_CODES|error_codes|ErrorCode\(' comfy_cli 2>/dev/null | head -260
printf '%s\n' '--- supported platform declaration ---'
rg -n -A8 -B8 'Cross-platform|requires-python|Operating System|OS\.MACOS|class OS' README.md pyproject.toml comfy_cli/constants.py comfy_cli/utils.py 2>/dev/null | head -180Repository: Comfy-Org/comfy-cli Length of output: 38208 🤖 get_repo_knowledge executed:
Length of output: 2911 🏁 Script executed: #!/bin/bash
set -eu
printf '%s\n' '--- renderer implementation ---'
fd -t f -a 'renderer.py' comfy_cli
sed -n '1,280p' comfy_cli/output/renderer.py
printf '%s\n' '--- error registry definition and ws_timeout consumers ---'
sed -n '1,210p' comfy_cli/error_codes.py
rg -n -A12 -B12 'as_discover_rows|load_error_codes|error_codes\[|get_error|registry|hint' comfy_cli/output comfy_cli | head -260Repository: Comfy-Org/comfy-cli Length of output: 39546 Restrict The pretty timeout message is printed on every supported platform. The shared Append 🤖 Prompt for AI AgentsThere was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ⚪ Nit — |
||
| ) | ||
| details = {"timeout": timeout} | ||
| prompt_id = _submitted_prompt_id(execution) | ||
|
|
@@ -677,7 +679,7 @@ def execute( | |
| renderer.error( | ||
| code="ws_timeout", | ||
| message=f"WebSocket timed out after {timeout}s waiting for server response.", | ||
| hint="re-run with a larger --timeout (e.g. --timeout 300)", | ||
| hint="re-run with a larger --timeout (e.g. --timeout 300); if the machine slept, check `comfy jobs status <id>`", | ||
| details=details, | ||
| ) | ||
| raise typer.Exit(code=1) | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -27,6 +27,7 @@ | |
|
|
||
| import json | ||
| import os | ||
| import time | ||
| import urllib.error | ||
| import urllib.parse | ||
| import uuid | ||
|
|
@@ -35,6 +36,7 @@ | |
| import typer | ||
| from rich.progress import BarColumn, Progress, TimeElapsedColumn | ||
| from rich.table import Column, Table | ||
| from websocket import WebSocketTimeoutException | ||
|
|
||
| from comfy_cli import execution_errors | ||
| from comfy_cli.caller import usage_source | ||
|
|
@@ -47,6 +49,13 @@ | |
|
|
||
| workspace_manager = WorkspaceManager() | ||
|
|
||
| # Upper bound on how long a single ``recv`` blocks before ``watch_execution`` | ||
| # regains control and re-checks the wall clock. Deliberately independent of the | ||
| # user's ``--timeout`` (which can be large for long jobs) so that even a big | ||
| # silence budget can never leave us blocked inside one ``recv`` across a system | ||
| # sleep — see the wall-clock backstop in ``watch_execution``. | ||
| _RECV_POLL_SECONDS = 30 | ||
|
|
||
|
|
||
| def _safe_close(execution: WorkflowExecution) -> None: | ||
| """Best-effort WebSocket close on cancellation.""" | ||
|
|
@@ -307,9 +316,35 @@ def queue(self): | |
| def watch_execution(self): | ||
| if self.ws is None: | ||
| raise RuntimeError("watch_execution called before the websocket was connected") | ||
| self.ws.settimeout(self.timeout) | ||
| # ``recv`` blocks on a socket timeout enforced against a MONOTONIC clock, | ||
| # which stops advancing while the machine is asleep (a laptop lid closed | ||
| # mid-run). On wake the connection is frequently dead, yet that timeout | ||
| # has under-counted the sleep, so it never fires and the loop hangs | ||
| # indefinitely instead of bailing out. Mirror the cloud waiter | ||
| # (``ComfyClient.wait_for_completion``): bound each ``recv`` to a short | ||
| # poll so the loop regains control regularly, and enforce the | ||
| # ``--timeout`` silence budget against the WALL clock, which DID advance | ||
| # across the sleep. A wake with a dead connection then aborts promptly | ||
| # (the caller reports ``ws_timeout`` — the server job is resumable via | ||
| # ``comfy jobs status``) rather than stalling for as long as the machine | ||
| # slept. | ||
| poll = min(self.timeout, _RECV_POLL_SECONDS) if self.timeout else self.timeout | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟢 Low — Because the budget is only evaluated when a poll expires, the effective silence timeout is rounded up to the next multiple of 30s: |
||
| self.ws.settimeout(poll) | ||
| last_activity = time.time() | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Medium — Basing the silence budget solely on |
||
| while True: | ||
| message = self.ws.recv() | ||
| try: | ||
| message = self.ws.recv() | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟢 Low — The 30s cap bounds each underlying socket read, not the |
||
| except WebSocketTimeoutException: | ||
| # No frame this poll interval. Give up only once the silence | ||
| # budget has elapsed in REAL time — a monotonic timer frozen by | ||
| # a sleep can no longer keep us waiting past it. | ||
| if time.time() - last_activity >= self.timeout: | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Medium — Charging suspend time against the silence budget also kills runs whose connection is fine: for the primary local/loopback case both processes freeze and resume together, so the TCP connection survives the sleep, and if the job happens to be in a quiet phase on wake the first poll expiry sees |
||
| raise | ||
| continue | ||
| # Any frame — even a non-text control/binary frame — proves the | ||
| # connection is live, so it resets the silence budget exactly as the | ||
| # per-``recv`` socket timeout used to. | ||
| last_activity = time.time() | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Medium — The comment above is wrong for control frames: |
||
| if not isinstance(message, str): | ||
| continue | ||
| try: | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🟢 Low — This handler also catches the timeout raised by
execution.connect(), where nothing was ever queued:_submitted_prompt_id(execution)a few lines below returnsNoneanddetailscarries noprompt_id, yet the new text (and thehintin the envelope) tells the user to check a job that does not exist. Gate both strings on the already-computedprompt_id is not None, which would also let them interpolate the real id instead of the literal<id>placeholder. Raised by 1 of 6 reviewers (claude-opus-5-thinking-max edge-case).