Add failure-path tests for data download and parquet cache - #193
Merged
Conversation
Agent-Logs-Url: https://github.com/ch55secake/hyperion/sessions/b4508b35-68ae-4fa0-b65b-4cdc33d70e25 Co-authored-by: ch55secake <87881861+ch55secake@users.noreply.github.com>
…ariable rename Agent-Logs-Url: https://github.com/ch55secake/hyperion/sessions/89f9a338-8869-4d7b-9ab9-80291e1f5f48 Co-authored-by: ch55secake <87881861+ch55secake@users.noreply.github.com>
Owner
|
@copilot typecheck failing |
Agent-Logs-Url: https://github.com/ch55secake/hyperion/sessions/ec7d0e21-160a-46b9-bef8-ea5b54ec93d1 Co-authored-by: ch55secake <87881861+ch55secake@users.noreply.github.com>
Contributor
Author
Fixed in ddddd15. Three changes to
All 36 tests pass and |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
StockDataDownloaderhad no tests for failure scenarios and no production-side handling for corrupted or schema-drifted parquet cache files, leaving the pipeline vulnerable to silent crashes on common real-world failures.Changes in this pull request
Production (
src/data/stock_data_downloader.py)REQUIRED_COLUMNS: frozenset— the five price columns yfinance always returns_load_cached_parquet(path, symbol) -> (df | None, needs_refresh)static helper that consolidates all cache validation:pd.read_parquetin atry/except— corrupted/truncated files trigger re-download instead of propagating an exceptionREQUIRED_COLUMNS - set(df.columns)) — missing columns trigger re-download_download_singlenow delegates to the helper, eliminating thedf = Noneguard patterntytypecheck errors: addcastto thetypingimport; addassert df is not Nonein the cached path to narrowpd.DataFrame | Nonebefore the dict assignment; usecast(pd.DataFrame, ticker.history(...))andassert isinstance(df, pd.DataFrame)after the retry loop to narrow the union type produced by yfinance's untyped returnTests (
tests/data/test_stock_data_downloader.py)TestDownloadFailurePaths— new class covering all four production failure modes:test_partial_failure_returns_successful_data—@pytest.mark.parametrizeover (1/3, 2/4, 3/3) failing tickers; asserts surviving tickers are returned and failures are tracked infailedtest_all_tickers_fail_returns_empty_dict— all tickers raise; assertsresult == {}with no crashtest_corrupted_parquet_triggers_redownload— garbage bytes written to cache path; asserts yfinance is called and symbol is inresulttest_schema_drift_triggers_redownload— cache missingClose/Volume; asserts re-download fires and full schema is present in result