Skip to content

Add failure-path tests for data download and parquet cache - #193

Merged
ch55secake merged 5 commits into
mainfrom
copilot/add-failure-path-tests
Apr 20, 2026
Merged

Add failure-path tests for data download and parquet cache#193
ch55secake merged 5 commits into
mainfrom
copilot/add-failure-path-tests

Conversation

Copilot AI commented Apr 19, 2026

Copy link
Copy Markdown
Contributor

Description

StockDataDownloader had no tests for failure scenarios and no production-side handling for corrupted or schema-drifted parquet cache files, leaving the pipeline vulnerable to silent crashes on common real-world failures.

Changes in this pull request

Production (src/data/stock_data_downloader.py)

  • Add module-level REQUIRED_COLUMNS: frozenset — the five price columns yfinance always returns
  • Extract _load_cached_parquet(path, symbol) -> (df | None, needs_refresh) static helper that consolidates all cache validation:
    • Wraps pd.read_parquet in a try/except — corrupted/truncated files trigger re-download instead of propagating an exception
    • Checks for schema drift (REQUIRED_COLUMNS - set(df.columns)) — missing columns trigger re-download
    • Staleness check (existing behaviour, now co-located)
  • _download_single now delegates to the helper, eliminating the df = None guard pattern
  • Fix ty typecheck errors: add cast to the typing import; add assert df is not None in the cached path to narrow pd.DataFrame | None before the dict assignment; use cast(pd.DataFrame, ticker.history(...)) and assert isinstance(df, pd.DataFrame) after the retry loop to narrow the union type produced by yfinance's untyped return

Tests (tests/data/test_stock_data_downloader.py)

  • TestDownloadFailurePaths — new class covering all four production failure modes:
    • test_partial_failure_returns_successful_data@pytest.mark.parametrize over (1/3, 2/4, 3/3) failing tickers; asserts surviving tickers are returned and failures are tracked in failed
    • test_all_tickers_fail_returns_empty_dict — all tickers raise; asserts result == {} with no crash
    • test_corrupted_parquet_triggers_redownload — garbage bytes written to cache path; asserts yfinance is called and symbol is in result
    • test_schema_drift_triggers_redownload — cache missing Close/Volume; asserts re-download fires and full schema is present in result

Agent-Logs-Url: https://github.com/ch55secake/hyperion/sessions/b4508b35-68ae-4fa0-b65b-4cdc33d70e25

Co-authored-by: ch55secake <87881861+ch55secake@users.noreply.github.com>
…ariable rename

Agent-Logs-Url: https://github.com/ch55secake/hyperion/sessions/89f9a338-8869-4d7b-9ab9-80291e1f5f48

Co-authored-by: ch55secake <87881861+ch55secake@users.noreply.github.com>
@ch55secake
ch55secake marked this pull request as ready for review April 20, 2026 20:54
@ch55secake ch55secake changed the title [WIP] Add failure-path tests for data download and parquet cache Add failure-path tests for data download and parquet cache Apr 20, 2026
@ch55secake

Copy link
Copy Markdown
Owner

@copilot typecheck failing

Agent-Logs-Url: https://github.com/ch55secake/hyperion/sessions/ec7d0e21-160a-46b9-bef8-ea5b54ec93d1

Co-authored-by: ch55secake <87881861+ch55secake@users.noreply.github.com>

Copilot AI commented Apr 20, 2026

Copy link
Copy Markdown
Contributor Author

@copilot typecheck failing

Fixed in ddddd15. Three changes to stock_data_downloader.py:

  1. Added cast to the typing import
  2. assert df is not None in the cached path (before the dict assignment) — ty couldn't narrow from the _load_cached_parquet return type
  3. cast(pd.DataFrame, ticker.history(...)) + assert isinstance(df, pd.DataFrame) after the retry loop — ty saw yfinance's untyped return as Unknown and was unioning it with the outer-scope DataFrame | None

All 36 tests pass and ty check src/ is clean.

@ch55secake
ch55secake merged commit 6a062e2 into main Apr 20, 2026
8 checks passed
@ch55secake
ch55secake deleted the copilot/add-failure-path-tests branch April 20, 2026 21:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Testing: Add failure-path tests for data download and parquet cache

2 participants