Implement the FileToLink improvement plan (H1-H10, M1-M14, L1-L10) + review hardening - #50
Open
fyaz05 wants to merge 31 commits into
Open
Implement the FileToLink improvement plan (H1-H10, M1-M14, L1-L10) + review hardening#50fyaz05 wants to merge 31 commits into
fyaz05 wants to merge 31 commits into
Conversation
…H1, H3, L8) H1 — security dependency pass: - pyproject.toml declares 8 exact-pinned runtime deps (was 10 unpinned) - aiohttp 3.14.3 (OSV-clean; the 3 Jan-2026 CVEs), uvloop 0.22.1 (py3.13), tgcrypto -> tgcrypto-pyrofork 1.2.8 (maintained crypto backend) - requirements.txt kept as a generated export for the Docker path - pip-audit verified clean locally; enforced in CI H3 — quality.yml gates (replaces docker-only CI; docker build now runs after quality passes): - ruff check + format, mypy (permissive baseline), pytest unit tier, pip-audit, bandit -ll, vulture dead-code gate (whitelist.py), dependency-count gate (>8 direct deps fails the build) - .pre-commit-config.yaml mirrors the local workflow L8 — repo hygiene pack: - Dockerfile: non-root USER, HEALTHCHECK -> /health, git kept for H9 updates - Makefile (format/lint/test/audit/run), .dockerignore (image no longer ships .git/README/tests), dependabot.yml (pip + github-actions) - SECURITY.md, CONTRIBUTING.md - AGENTS.md rewritten: new conventions (tg_call, preflight chain, HTML escaping, budgets), command table generated from bot/registry.py - config_sample.env: every new env var annotated (PRIVATE_MODE, ENABLE_SHELL, ENABLE_LEGACY_LINKS, FILE_TTL_DAYS, worker pools, MAX_CONCURRENT_STREAMS, touch buffer, LOG_LEVEL/LOG_FORMAT, ...)
The boot path interpolated UPSTREAM_REPO/UPSTREAM_BRANCH into a shell=True command chain and ran 'rm -rf .git', 'git reset --hard' and mutated global git config — a live injection vector at every container boot, and a failed update could leave a half-wiped tree. - argv-list subprocess (shell=False); remote URL stays an argv element - 'git pull --ff-only' replaces the init/commit/reset dance - strictly best-effort: failures log and keep the current code running - no-ops cleanly when git is absent or the tree is not a repo - thunder.sh reduced to boot orchestration (update failure no longer blocks boot); /restart marker flow unchanged
…4, L1-L7, L9-L10)
P2 — core hygiene (H4, H5, H10):
- utils/safe_call.py: tg_call helper + reply/send/edit/delete/answer_safe
wrappers; all ~90 inline try/except-FloodWait pairs migrated (budget:
custom_dl streaming loop + canonical ingest loop only, as planned)
- dead code purged: dl.html + its render branch, tokens.list_tokens/
get_user, bot_utils.notify_ch, rate_limiter.get_queue_status,
file_properties.get_fids, canonical_files.touch_file_record +
get_file_by_message_id (+ db twin), on_fallback_message stream path,
Var.MULTI_CLIENT, MSG_STATS_ERROR, dead client_max_size
- three drifted media maps unified into utils/media_types.py (H4c)
- speedtest-cli removed (archived upstream, H5a); shortener rebuilt on
aiohttp, cloudscraper dropped (H5b; curl_cffi optional extra)
- H10: LOG_LEVEL/LOG_FORMAT, shared redact_secrets(), access-log
middleware with sha256-hashed file tokens, /log uploads redacted +
capped at 45 MiB
P3 — runtime correctness (H6, H7, H8):
- rate limiter: worker pool (EXECUTOR_WORKERS), charge-at-exec (the old
code double-charged: enqueue + executor), requeue-on-FloodWait with
attempt counter (long FloodWaits no longer stall every user), global
RPS token-bucket breaker (burst 2x), bounded structures + 5-min sweep,
limiter occupancy in /stats
- flag_cache.py: TTL+LRU cache in front of ban/authorization checks,
FAIL-CLOSED on DB errors (a Mongo outage no longer un-bans everyone),
invalidated by admin mutators; OWNER_ID missing now refuses boot
- H8: Mongo timeoutMS=5000, tg_call RPC budgets (transfers unbounded),
shortener/keepalive 10 s, last sync psutil call moved to a thread
P4 — robustness & UX parity (M1-M14):
- M6: vars.py validates ALL config problems together with named vars,
config.env.local layering, bounds enforcement
- M1: bot/registry.py single command registry -> menu (owner-only
hidden, 256-char truncation), /help section, AGENTS.md drift test
- M3: /health endpoint (zero deps); keepalive repointed off the
GitHub-redirecting root onto /health with status checking
- M2: typed player page (video/audio/image/non-media cards) from mime
type, CWD-independent Jinja loader, noindex/nofollow, noscript hint
- M7: user-facing surfaces are HTML with html.escape()d file names,
titles and ban reasons; markdown escaper retired
- M8: atomic token activation (find_one_and_update conditioned on
activated != True — exactly one concurrent /start wins) +
/activate/{token} web route for shorteners
- M4a: broadcast on a 4-worker queue pool with Mongo cursor streaming,
200 ms pacing, progress every 25 sends, cancel preserved
- M4b: batch on a 5-worker pool, order-preserving results, skipped vs
failed counters, 30+2n deadline, MAX_BATCH_FILES enforced
- M5: shortener cache + singleflight, Bearer header auth, host
validation, https-only, no redirects
- M9: MAX_CONCURRENT_STREAMS env admission cap (503 + Retry-After kept)
- M10: self-healing vault records — unresolved vault message => record
deleted + 404 => next upload re-ingests; Content-Length verified
against the actual vault media (stuck-at-99% fix)
- M11: callback panic-isolation guard with error-ID owner notification;
close_panel permission check; catch-all answered (was already present)
- M12: unified preflight chain (banned -> private-mode -> token ->
force-sub -> shortener) + PRIVATE_MODE allowlist; /start keeps the
activation flow reachable; utils->bot inversions broken via client
params (lazy fallback keeps compatibility)
- M13: bounded shutdown drain (30 s, work_loads-based) + aggregated
teardown errors, non-zero exit on failure; restart-marker preserved
- M14: touch buffer capped with drop-on-overflow counter, env-tunable
1-60 s flush, single BulkWrite per cycle
P5 — polish (L1-L7, L10):
- L1: ENABLE_LEGACY_LINKS flag (default on; off => 410) + legacy page
cache (repeat /watch views no longer hit Telegram)
- L2: optional FILE_TTL_DAYS TTL index with backfill; expiry note in
link messages only when enabled
- L3: /status gains DC id, inflight, touch-buffer stats, no-store
- L4: 32-hex public hash for new uploads; 20-hex links stay valid
- L5: session files chmod 0600 after startup
- L6: RFC 5987 filename* with ASCII fallback; 416 + Content-Range kept
- L7: preconnect hints, inline fallback stylesheet + noscript so the
player page stays usable with CDNs blocked
- L10: /shell behind ENABLE_SHELL kill-switch (default off) + command log
…n tier - conftest.py: hermetic env bootstrap (Thunder.vars validates at import) - unit tier (pytest -m unit, asyncio_mode=auto, no network/Mongo): human_readable, time_format, media_types unification, safe_call FloodWait/timeout semantics, flag_cache TTL/LRU/fail paths, rate limiter window math + breaker + sweep, canonical hash dual-length + merge precedence, stream_routes parsing/range/disposition, shortener registry + host validation, secret redaction, config validation (subprocess: all problems reported together, OWNER_ID boot-fail), registry menu/help/AGENTS.md drift checks - coverage gate --cov-fail-under=35 as the starting bar (raise to 80 on core modules next) - tests/integration: testcontainers MongoDB (ingest-claim + atomic token activation), TEST_INTEGRATION=1 gated, skipped cleanly without Docker (mirrors ThunderGo's build-tag-gated tier, but opt-in runnable in CI)
…7 -> 7 blocks Remaining 7 inline FloodWait references are the deliberate ones: - safe_call.py (the helper itself, 2) - custom_dl.py streaming pool loops (2, explicitly allowed by plan) - rate_limiter.py requeue-on-FloodWait executor logic (1) - canonical_files.py re-raise into the ingest retry loop (1) - broadcast.py classification of exhausted FloodWait as failed send (1)
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Real bugs found by the recheck pass: - shortener: allow_redirects=False on ClientSession() raises TypeError at runtime (aiohttp rejects it there) and silently disabled the M5 shortener; redirects now enforced per-request on all 4 plugin calls - database: missing await on AsyncCollection.aggregate() broke the duplicate-user dedup path (async-for over a coroutine raises TypeError) Audit gaps closed (plan fidelity): - H1: commit uv.lock (hash-pinned, 70 pkgs, uv sync --frozen verified on py3.13.15) + pip-audit in dev group + uv lock --check CI gate - H4b/L8 hygiene: remove committed runtime log Thunder/logs/bot.txt, gitignore Thunder/logs/ (+ fix log.text -> log.txt) - L8: __version__ now injectable via APP_VERSION env (default 2.2.0, aligned with pyproject; /status reads it) - quality.yml: docker publish restricted to refs/heads/main so feature pushes can never overwrite fyaz05/thunder:latest - mypy made BLOCKING: all 44 errors fixed (2 real bugs above + typed dict/float annotations + defensive list-narrowing + targeted stub-union ignores with comments); Makefile lint no longer || true - 45 tracked files restored to 644 (stray 755 mode bits) Gates on final tree: ruff check+format PASS, mypy 0 errors (35 files), 103 unit tests PASS (py3.13.15, locked deps), vulture PASS, bandit medium+ = 0, pip-audit clean, dep count = 8, uv lock --check PASS
CRITICAL - rate_limiter: deferred requeue busy-spun with ZERO yield points (Event.wait fast-path + uncontended Lock on py3.13) -> whole event loop froze up to 300s; now parks the pool (event clear + call_later timer), re-wakes on enqueue, timer cancelled at shutdown - quality.yml: invalid YAML (unquoted ':' in step name) killed every gate; Makefile recipes had spaces not tabs (all targets dead) HIGH - callbacks: close_panel compared clicker to the BOT's id -> every non-owner Close button rejected; now allows panel reply target / private-chat peer (+ owner) - ban gate was fail-OPEN: is_user_banned swallowed Mongo outages into None (negative-cached 5 min) -> raise_on_error=True for the flag loader; ban/unban now invalidate the flag cache - HTML injection (M7 gaps): file_name in link messages, first_name in new-user log, chat_title/source_info in 5 stream.py sites escaped - custom_dl: mid-stream FloodWait restarted from the original offset, re-sending delivered bytes (corrupted downloads); now resumes at chunk_offset + chunks_done; get_message via bounded tg_call - stream_routes: Range end >= file_size now clamps per RFC 7233 (was a hard 416, breaking resume clients) - integration tier was dead twice over: wrong testcontainers import + missing dep + unit cov threshold; testcontainers[mongodb] declared, cov flags moved out of addopts - access-log redaction hashed m.group(0)[:-len(m.group(0))] == "" (constant pseudonym for every file) - CI now installs the hash-pinned uv.lock (uv sync --frozen) and audits the locked env incl. transitives; permissions hardened MEDIUM - touches added during an in-flight flush are re-armed; failed bulk-flush merges back instead of dropping increments - canonical validation RPC errors keep the cached record instead of re-copying into BIN (vault churn) - broadcast: strong task refs, return_exceptions, finally-cleanup of workers/status/registry; completion only on success - shortener: init lock (session leak), singleflight resolves waiters on cancellation, Bitly host validation, session closed at shutdown - requeued requests no longer re-charge the user window; breaker token consumed only at exec; cancelled workers release counts - flusher lifecycle, force_channel negative-cache, token consume expiry guard, tokens -> aware UTC, update.py missing return + credential redaction + orphaned config backup recovery, startup teardown order (touch drain before db.close), umask 077, PRIVATE_MODE x TOKEN_ENABLED fail-fast, preflight unknown-gate warning, /log off-loop, /listauth batched, curl_cffi floor 0.15.0 (CVE-2026-33752) LEANNESS - dead _cache_by_message_id map + get_file_by_message_id removed; Var.LOG_LEVEL/LOG_FORMAT dead attrs removed; dead skip param and RPCError branch removed; help panel reuses build_help_text; README /speedtest residue removed; dep-count gate added to make audit Tests: 107 passed (+4 regression: Range clamp, singleflight, park x2); ruff/mypy/vulture/bandit/pip-audit/uv-lock gates all green.
- shortener: replace substring host matching with parsed-hostname exact/suffix match (_host_matches) across all 4 HTTP plugins; lookalikes like 'bitly.com.evil.com' or 'evil.com/bitly.com' no longer select a provider plugin (CodeQL high: py/incomplete-url-substring-sanitization) - /activate: strict shape check (_is_activation_token) on the token before building the t.me redirect -- tokens are token_urlsafe(32), 43 URL-safe chars; malformed input now 400s without a DB roundtrip (CodeQL medium: py/url-redirection) - tests: +21 regression tests (host matching incl. adversarial lookalikes, activation token shape incl. injection payloads)
CodeQL py/url-redirection still tracked the shape-validated token into the Location header; build the deep link via quote_plus(safe='') so the token can only occupy the query-value slot. No-op for valid tokens (already URL-safe); +2 tests.
CodeQL py/url-redirection models only '+ right-operand' as a sanitizer (f-strings and quote_plus are not barriers -- verified against UrlRedirectCustomizations.qll on github/codeql@main). Build the deep link so the constant 'https://t.me/' prefix is provably outside user control; output is byte-identical for valid tokens.
Behavioral (verified by repro/code-reading): - caption commands: /link & /dc parsed msg.text (None for captions under pyrofork's caption-matching filter) -> AttributeError dead air; now text-or-caption - channel-ban gate: per-post DB find_one, no cache, no invalidation; now flag-cached with add/remove invalidation (fail-open kept deliberately: leave_chat on a Mongo outage would be destructive) - legacy route error masking: get_file_info swallowed every exception into an unread 'error' key -> transient outages surfaced as 404; now propagates (route ladder maps 404/500 correctly) - markup injection (pyrofork DEFAULT = markdown-then-HTML, so html.escape alone is insufficient): /dc templates converted to HTML + escaped + ParseMode.HTML pinned; force-sub prompt escaped; ban reason escaped; file-request sends pin ParseMode.HTML - /restart: os.execv skipped the M13 touch-buffer drain, losing view counts; now drains before execv - access-log middleware logged nothing for non-HTTP 500s (response unbound NameError swallowed); response now initialized to None - integration fixture: env+reload never rebound the frozen Var/db copies -> tier could not pass; now binds Database directly and rebinds module handles, restoring state after - Docker HEALTHCHECK: naive PORT parse broke on inline comments in the shipped sample config; now comment-stripping - GLOBAL_RPS_LIMIT dead without GLOBAL_RATE_LIMIT: boot warning (M6) - token_col: compound (user_id, activated, expires_at) index added - MSG_TOKEN_FAILED: error_id kwarg was silently dropped (no slot) - update.py: rejects leading-dash UPSTREAM_REPO/BRANCH (argv hygiene) - _validate_short_url: trailing-dot FQDN now tolerated (parity with _host_matches); import_plugins glob sorted; get_user deduped; force-channel button reuses cached get_force_info; render_template reuses quote_media_name; breaker comment states real invariant Leanness (dead code, vulture-verified): - deleted whitelist.py (entirely stale; vulture passes without), ShortenerError, module-level request_executor, duplicate status_options handler, _REDACT_SEGMENTS, production-dead mime_for Infra: - Dockerfile: drop build-essential/libssl-dev (all wheels, ~150MB) - image tags: + SHA tag for rollback; .dockerignore: cache dirs - pre-commit ruff rev v0.8.4 -> v0.16.6 (matches lock); Makefile audit now audits the locked env (CI parity); dependabot uv.lock note; README: dead speedtest section removed, docker run now mounts config.env (documented flow was broken) Gates: ruff/format clean, mypy 0 (35 files), 130 tests PASS, vulture (whitelist-free) clean, bandit clean, uv lock --check PASS
…rmed dev extra - C-1: AsyncMongoClient now uses tz_aware=True; pymongo's default returns naive UTC datetimes, and comparing them against datetime.now(UTC) in tokens.consume() raised TypeError on EVERY real activation -- the flagship M8 /start <token> path was dead at runtime - lost CAS races now distinguish 'already' (winner activated) from 'invalid' (expired / missing expires_at) instead of always saying 'already' - pyproject: 'testcontainersongodb]>=4.13' -> 'testcontainers[mongodb]>=4.13' (invalid PEP 508; fresh 'uv lock' hard-failed on it) - generate() no longer returns '' on an unreachable path (honest raise)
…d logs Self-heal safety: - transient Telegram failures (FloodWait exhaustion, timeout, transport) now raise TelegramUnavailable -> 503 + Retry-After; they no longer masquerade as FileNotFound, which made a brownout DELETE every vault record requested during the outage - self-heal replacement preserves the existing public_hash (rewriting a legacy 20-char hash broke the 'legacy links stay valid forever' contract) - stream_file fetches its target once (no re-fetch per FloodWait retry) and bounds total FloodWait sleep (60s) instead of pinning the handler - safe_call caps a single FloodWait sleep at 30s (H8 budget honesty) Gates / access control (fail-closed): - preflight: unknown gate ids REJECT (were warn-and-continue = skipped security check); gate presets GATES_STANDARD/START/INFO added and common.py now routes through them (was hand-rolled per handler) - PRIVATE_MODE denies unattributable senders (from_user None); channel handler refuses to mint public links in PRIVATE_MODE - token deep-link uses quote_plus like the hardened /activate builder Rate limiter: - the immediate path now consumes a breaker token; a dry bucket falls through to the queue (bursts are shaped, not shed) -- the old top-shed also claimed queue-full, misleading users - breaker gate unified on breaker.rate > 0 (GLOBAL_RPS_LIMIT-only setups) - removed write-only user_queue_counts (+ its cancel-path leak edge) - requeue-vs-rotation insertion policy documented Boot / shutdown: - boot failures exit 1 (restart policies can actually fire) - background tasks tracked in one list (no locals().get bookkeeping); index-ensure task cancelled+awaited at shutdown; batched 10s wait - deprecated get_event_loop().time() -> get_running_loop() Config: - config.env.local really overrides config.env now (load_dotenv override=False had inverted the documented precedence) - str_to_int_set surfaces junk tokens (collect-all-errors) Logging / misc: - access log redacts legacy id-first capability hashes and /activate tokens; control chars escaped (log forging via %0A neutralized) - redact_secrets covers API_HASH, session strings, ?start= activation tokens - MSG_SHELL_OUTPUT_CAPTION fixes a format() KeyError that swallowed large /shell outputs; render_page no longer logs the capability hash - shortener URLs html-escaped into the HTML links message - update.py allows only https/http/git/ssh schemes (blocks ext::) - leanness: dead touch_file_record, requested_action, task_done, peek/ occupancy, unreachable branches, dead getattr defaults, duplicate disk syscalls, legacy-cache unique_id all removed
…+ integration tier - Makefile recipes indented with hard TABs (spaces made every target fail with 'missing separator') - Dockerfile installs requirements.lock (uv export of uv.lock, --require-hashes): transitives were previously resolved fresh and unpinned on every build, so CI's supply-chain guarantees never reached the shipped image; git dropped from the image (self-update no-ops without .git by design -- replace the image instead) - CI: push trigger 'branches: ain]' corruption fixed to [main] - CI: requirements.txt<->pyproject and requirements.lock<->uv.lock sync gates (two sources of truth must agree) - CI: integration tier (real MongoDB via testcontainers) now actually runs -- the atomicity guarantees were never executed in CI - CI: concurrency group + job timeout-minutes - README: SLEEP_THRESHOLD 600 / GLOBAL_RATE_LIMIT False (vars.py ground truth); auto-update note (needs a git checkout) - AGENTS.md documents the requirements.lock flow
…gressions - test_preflight: preset ids resolve in the registry, documented order, unknown gate id rejects fail-closed, None-vs-False return contract - test_config_env_layers: config.env.local overrides config.env; real environment still wins (subprocess, hermetic) - test_canonical_files: replacement merge preserves legacy public_hash - test_custom_dl_exceptions: transport errors -> TelegramUnavailable, genuine absence -> FileNotFound (self-heal safety contract) - updated flag-cache and int-set tests to the leaner/fail-closed behavior
Owner
Author
|
All audit findings fixed and green: 1 critical (tz-naive token activation crash), 12 majors (self-heal record deletion, fail-open gates, State: 159 unit + 3 integration tests, ruff, mypy, bandit, pip-audit all green in CI on head |
…ntegration tests The previous fix passed timeoutMS=120_000 to update_many, which pymongo 4.18 does not accept (no per-op CSOT override on CRUD) -- every boot with FILE_TTL_DAYS>0 raised TypeError, aborted ensure_indexes and silently skipped all unique indexes: the exact failure mode the fix was meant to remove, masked in CI because tests only exercised the TTL=0 path. Replace the single COLLSCAN update_many with an _id-paged micro-batch migration (every statement an index seek inside the client-wide 5s budget), bounded per boot and resumable via a migration marker; scope the TTL conflict handler to IndexOptionsConflict (code 85) and treat ExecutionTimeout as 'build continues server-side' instead of dropping a live build. Add integration coverage for the full TTL lifecycle (first boot backfill, TTL-change recreate with uniques intact, steady state, and real row expiry).
…ogs, add route-ladder regression tests - AppRunner(access_log=None): aiohttp's own access logger still emitted raw %r request lines on a latent sink (inert today only because the logger has no handlers; any future basicConfig() would have leaked unredacted paths). - canonical stream warnings no longer log the capability hash alongside record/vault sizes; /log would have handed out a link fingerprint. - unit tests pin the delivery ladder: TelegramUnavailable -> 503 with Retry-After and NO record deletion; true absence / media-less vault message -> self-heal + 404; admission slot released on every path.
Leftovers the fix session missed, all verified unreachable or write-only:
- config_parser try/except that cannot raise (digit filter + or 0)
- registry Command.hidden speculative knob ('none today')
- stream.py batch worker: orphan task_done() (no queue.join exists) and
the dead getattr(Var, 'BATCH_WORKERS', 5) fallback
- decorators.py: unreachable empty-token guard (generate() raises, never
returns empty)
- shortener.py: four dead getattr(Var) fallbacks (vars.py validates all)
- messages.py: unused MSG_SHELL_OUTPUT constant
- tests: immediate-path breaker consumption + dry-bucket fall-through,
consume() status ladder incl. corrupt token without expires_at
…tency, doc truth - vars.py honours THUNDER_SKIP_CONFIG_FILES=1; conftest sets it so a developer's own config.env cannot leak optional knobs into the unit tier; the subprocess precedence tests opt back out explicitly - CI ruff scope now covers tests/ (was Thunder/ update.py only) - Makefile: all targets through 'uv run' (was mixed ambient/project envs), find-based clean (globstar is not POSIX sh) - requirements.txt header: the Docker image consumes requirements.lock; this file is the direct-dependency surface kept in sync by CI - README: document config.env.local precedence (FAQ claimed 'config.env Only')
The four sequential re.sub rules let the id-first rule re-match the 8-hex pseudonyms produced by the canonical rule (~39% of them end in two digits), double-hashing the same file and stamping a misleading "…" truncation marker on canonical /f/ links. One combined alternation pass pins exactly-once semantics; 10 unit tests cover every family (canon 20/32, watch, id-first, activate), the digit-tailed-pseudonym regression, and the control-char escape.
Untyped function bodies were previously unchecked; enabling the flag surfaces 5 real issues, all fixed at the annotation level: - process_single(status_msg) accepts None from the batch worker, where no per-file status message exists (existing call sites already None-guard) -> Message | None - get_readable_time() already int()-coerces internally and a test feeds it 90.9 -> accept int | float (the /status uptime caller passes float) - pyrogram exposes Client.username dynamically; read via getattr with fallback in /status, pin via explicit ignore at boot
Fail-closed gates reject messages with no attributable from_user (channel posts, anonymous admins); say so in the README token section and the AGENTS gate-chain notes.
Every comment in the branch diff trimmed to 0-2 lines carrying only the why/invariant/pitfall (plan tags H/M/L kept). No code changes: verified by AST comparison (docstrings stripped) on all 47 touched .py files; Makefile tabs, TOML/YAML parse, and the full local gate suite unaffected.
fyaz05
changed the title
SLEEP_THRESHOLD=600 made pyrofork auto-sleep every FloodWait <= 600s INSIDE the RPC, so tg_call's 30s wait_for budget turned recoverable FloodWaits into TimeoutErrors and its except FloodWait branch was dead for all realistic waits. Default now 10s (pyrofork's own default), kept well below TG_RPC_TIMEOUT_SECONDS so auto-slept waits complete inside the budget; longer waits surface to tg_call's bounded retry. config sample comment corrected (threshold, not sleep time). Also on this path: - client.start / StreamBot.start: explicit 90s budget (session bootstrap outlives the lightweight-RPC budget); dropped the sleep(2) stagger that applied to exactly one client - boot: executor tasks registered before the web-boot try so a later boot failure cancels live workers (was: registered after site.start(), escaping the cancellation path) - rate_limiter: requeues always append (front-insertion was vestigial: every caller passes delay>0, and _process_one rotates deferred items to the back anyway); MAX_REQUEST_ATTEMPTS comment corrected (breaker/ user-limit requeues are not counted by it) - notify_own: dropped the dead try/except around gather(return_exceptions) and the stale send_with_flood_wait name - flag_cache.sweep: sync (it never awaited anything)
The idfirst rule required a trailing slash, but the catch-all route serves the no-filename legacy URL (/AbCdEf12345): its bare 6-char hash + message id is the entire link credential, and it reached the access log (and /log uploads, which have no 6-char-hash pattern) unredacted -- defeating the hash-fingerprinting guarantee 78c02e5 established for canonical stream warnings. Anchor now (?=/|$); regression tests cover the bare hash-first and id-only segments.
…eltas
- backfill: pages now filter on last_seen_at {$exists: false} and the
resume cursor is persisted per batch. Before: last_id was local, so a
vault >= 50k docs rescanned the first 50k every boot and never reached
the rest (their TTL window stayed 'forever'); an exact-multiple page
never triggered the done-marker. Convergence is pinned by unit tests
with a fake collection (50.6k docs across two boots).
- _create_file_ttl_index: the post-drop-85 recreate also catches
OperationFailure (a concurrent recreate must not abort the remaining
unique ensures, per the method's own contract)
- ensure_indexes: unique/plain ensures go through a best-effort helper
(ExecutionTimeout on a large index build aborted every ensure after it)
- touch buffer: pending entries carry (reuse_delta, seen_delta) so N
touches of one hash flush as N increments instead of a merged $inc: 1
(seen_count was silently undercounting /status and admin metrics)
- tokens: allowed() delegates to db.is_user_authorized instead of a
verbatim duplicate; unreachable try/excepts around cursor-returning
find() calls removed; ban-cache invalidation made unconditional on
both user and channel paths
- canonical record lookup: raise_on_error=True again (the =False call
made a Mongo transport failure indistinguishable from absence), with
a _fetch_file_record wrapper mapping failures to 503 + Retry-After --
brownouts used to return a cacheable 404, breaching the 'transport
errors -> 503' self-heal contract (regression test included)
- the 503 body no longer echoes str(e): TelegramUnavailable messages
embed the internal vault message id (CodeQL info-exposure sliver);
constant text, detail stays in the server log
- the two delivery routes' identical inner+outer ladders (~60 lines x2)
extracted into _admission_ladder/_route_ladder context managers;
preview routes share the outer ladder
- /status: dropped the 'inflight' field duplicating total_workload
- custom_dl: get_file_info_sync raises FileNotFound for media-less
messages (the {'error': ...} marker had no reader anywhere)
- parse_range_header: documented the deliberate 400-vs-RFC-ignore
choice for structurally-broken ranges
…d code - cancel_broadcast: owner_only guard like its sibling restart action (defense-in-depth: the button lives on the owner's /broadcast reply) - close_panel: delete_safe instead of raw message.delete() - batch summary: skipped files now counted (processed+failed < total with no explanation read as data loss to users) - broadcast: _PERMANENT_ERROR_REASONS.get(type(e), default) -- except matches subclasses, the exact-type dict lookup raised KeyError and killed the worker - logger: module-level hashlib import (per-log-line hot path) - commands: module-level import (no circular-avoidance justification, unlike its siblings); config_parser: dead config_file param removed; preflight test: pinned knob so the assertion is not a tautology
…ments curl_cffi was documented in pyproject, the shortener docstring and AGENTS.md as a Cloudflare escape hatch, but no code path ever imported it -- installing .[shortener-cf] pulled a large binary dependency that nothing used (leanness: deleting beats implementing a speculative fallback). uv.lock re-resolved: curl-cffi + cffi + pycparser removed. Floors/comments corrected while here: - testcontainers[mongodb] >= 4.15 (tests import testcontainers.community.mongodb, which only exists from 4.15; on 4.13/4.14 the ImportError guard silently skipped the whole tier) - pytest >= 8.4 (pytest-asyncio 1.4 requires it; lock already had 9.x) - aiohttp comment: the 3 fixed CVEs are from the Aug-2026 advisory cycle, not Jan-2026 (that release was 3.13.3 with different CVEs) - pyrofork pin annotated: upstream quiet since 2025-12 (drift risk)
- docker job: runs on PRs now (build only, no login, no push -- no secrets) and still publishes only from main pushes; before, the hardened Dockerfile was never exercised pre-merge, so regressions merged green and broke the post-merge publish. Added timeout-minutes - actions pinned to full commit SHAs (tag-pinned uses were the last mutable injection vector for the docker job holding DOCKER_TOKEN); dependabot's github-actions ecosystem keeps them current - uv pinned in both jobs: lock/export output format and gate semantics must not drift between commits - bandit: -c pyproject.toml (the [tool.bandit] section was inert without it); mypy/vulture/bandit/ruff scope now includes update.py (the security-sensitive self-update module was only ruff-linted) - Makefile: make lock (the three CI drift gates, previously local-green/CI-red), make integration, parity with the widened scopes - Dockerfile: dropped 'pip install --upgrade pip' -- the sole unhashed install; 3.13-slim's pip fully supports --require-hashes - CONTRIBUTING: uv sync --frozen setup (the old venv/pip instructions never installed uv, which every Makefile recipe requires), pre-commit wiring, make lock/integration documented - .dockerignore: uv.lock/requirements.txt/AGENTS.md excluded from the image (dead weight; requirements.lock stays -- it is COPY'd early); .gitignore trailing newline
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Full implementation of the improvement plan (H1-H10, M1-M14, L1-L10), hardened by an independent review cycle: 1 critical + 12 major findings, minors and leanness items — all fixed in 17 follow-up commits. Comments kept lean branch-wide.
What changed
config.env.localprecedenceuv export+--require-hashes) with CI requirements↔lock sync gates, single-pass access-log redaction + log-forging escape, scheme allowlist in self-update, activation-redirect hardeningVerification
159 unit + 3 integration tests (real Mongo), ruff, mypy (
check_untyped_defs), bandit, vulture, pip-audit,uv lock --check— green locally and in CI on every push.