feat: first-run experience and friendly error messages - #321
Merged
Conversation
luabida
changed the title
luabida
force-pushed
the
improve-impl
branch
2 times, most recently
from
August 23, 2026 21:10
59cfaf5 to
6f0d6c1
Compare
…ngue#320) * feat: OpenDataSUS catalog client and unified metadata layer - add pysus.api.saude client for dadosabertos.saude.gov.br: Next.js buildId discovery, catalog listing, full CKAN package metadata and resource downloads, with offline tests backed by captured fixtures - introduce the unified metadata architecture: MetadataBag with eight typed facets (identity, description, temporal, spatial, provenance, structure, access, quality), the MetadataExtractor protocol, and per-client extractors for FTP, DadosGov, DuckLake and Saude - wire .metadata/.ametadata() into every remote entity (clients, datasets, groups, files) via MetadataMixin; tabular local files compute structure metadata from their content - document both surfaces in docs/source/guides (saude.rst, metadata.rst) * feat(saude): add source-scoped dataset registry for OpenDataSUS * chore: ignore waypoints and roadmaps from repo * chore: include JSONL parsing * feat(saude): add DEMAS REST query path and JSONL file type - rest.py: EndpointSpec, fetch_swagger (cached), iter_rows (row-offset paginator), endpoints_from_swagger - JSONL: new file type in types.py, full class in extensions.py with load/stream/columns/rows, _detect_jsonl detector, factory registration - SaudeEndpointFile: BaseRemoteFile subclass for paginated DEMAS endpoints with SaudeEndpointFileExtractor for metadata - SaudeDataset._fetch_content() now returns CKAN groups + endpoint files - 48 new tests (37 saude + 11 extensions), full suite 1006 passed * feat(saude): integrate Saude into inventory + compare pipeline - PySUS.get_saude(): lazy SaudeClient init (no auth required) - Inventory._collect_saude(): walks SaudeGroup (CKAN) + SaudeEndpointFile (DEMAS REST) producing FileRecord objects with format=jsonl - ORIGINS/DOWNLOAD_PRIORITY: added 'saude' as 4th origin (fallback) - _ORIGIN_TO_CLIENT: registered 'saude' for collect() dispatch - Cross-origin identity works naturally via stem_of() normalisation - 8 new tests, full suite 1014 passed * feat(saude): wire Saude into sync engine (5.A + 5.B) - upload_file/download_raw_with_retry: widen type from FTPFile|APIFile to BaseRemoteFile (unblocks SaudeFile/SaudeEndpointFile) - _preconnect_adapters: add 'saude' to origin filter - _pick_source: add Saude as 3rd fallback (ftp > dadosgov > saude) - run(): collect Saude inventory, include in comparisons - _reprocess: Saude passes needs_token check naturally * feat(saude): add --saude-only flag (5.C) - SyncEngine.run(): add origins parameter to filter which origins are collected (defaults to all 4) - sync_clients.py: add --saude-only flag, passes origins=('ducklake','saude') - Fix test to match new run() signature * test(saude): integration tests for sync engine (5.E) * docs(saude): mark Stage 5 complete in progress tracker * feat(saude): DuckLake catalog integration (Stage 6) - DuckDataset.query(): fix state=NULL handling — Saude files are national-only (state IS NULL), IN clause never matches NULL; added OR state IS NULL to include them in queries - test_catalog.py: 6 new Saude integration tests — ensure_dataset creates entry, upsert_file with/without state, null-state query inclusion, multiple datasets, idempotent ensure_dataset - _catalog_rows already origin-agnostic (reads file.client.name), verified no changes needed - Stage 6 roadmap written * feat(saude): column metadata backfill + CHANGELOG (Stage 6.3) - CHANGELOG.md: document Stages 2-6 under 2.9.0 - pysus/api/saude/schemas.py: YAML schema loader with cached reads - pysus/api/saude/schemas/arboviroses.yaml: dengue/chikungunya/zika column definitions from SINAN documentation - sync.py _catalog_rows: apply YAML descriptions after link_columns when client is saude - 10 new tests for schema loader + apply_column_descriptions * docs(saude): tick all rollout checklist items, mark integration complete * docs(saude): commit stage-specific roadmaps * chore: remove roadmaps from repo, add roadmap*.md to .gitignore * style: black formatting * test(saude): improve coverage to 99% across saude module; fix sphinx CI - Add 100+ new tests for saude client, catalog, download, metadata, models, and resources modules - New test_metadata.py: full coverage of all 4 metadata extractors with all input shapes - catalog.py: 98% → 100% - client.py: 91% → 100% - download.py: 94% → 100% - metadata.py: 90% → 100% - models.py: 88% → 99% - resources.py: 99% → 100% - Fix sphinx CI: comment out automodule:: pysus.cli in api.rst since saude.py is intentionally uncommitted - Black/isort formatting fixes * fix(test): chdir to tmp_path for SaudeEndpointFile._download(None) test The test writes a relative path file when output=None; on CI the CWD is read-only, causing PermissionError. Fix by chdir to tmp_path. * fix(test): avoid CWD writes in SaudeFile._download(None) test The mock download_resource wrote to dest_dir=Path(".") which is read-only on Docker/CI. Write to tmp_path directly instead.
Phase 0.2 of the UX roadmap: - First-run welcome message on import (cache path, set_cache hint) - pysus.info() prints a table of all 27 datasets across FTP/Saude/DadosGov origins - Friendly error messages with actionable hints on all key exceptions: ConnectionError, AuthenticationError, ValidationError, DownloadError, FormatError now include hints about network, tokens, env vars, etc. - 9 new tests for info() and set_cache() - 1197 total tests passing
- Add pysus/api/progress.py with ProgressCallback wrapping tqdm - Auto-inject progress bars into download() when no callback given - Add pysus.enable_progress_bars() / disable_progress_bars() globals - 10 new tests, 1207 total passing
- Add as_dataframe, columns, dtypes params to PySUS.query() - When as_dataframe=True: downloads files, converts to Parquet, reads via DuckDB, returns single concatenated DataFrame - columns param selects subset, dtypes overrides column types - 5 new tests, 1212 total passing
- Add pysus/api/mappings.py with 140+ PT→EN column name mappings - Add pysus/api/flatten.py with flatten_json_columns() utility - Add pysus.to_english() and pysus.flatten_json_columns() top-level - 16 new tests, 1228 total passing
- Add pysus/api/columns.py with ColumnInfo dataclass and search_columns() - Searches YAML schemas (arboviroses) + SINAN typecast dictionaries - Exports pysus.search_columns() and pysus.ColumnInfo - Fix setup.cfg max-line-length to match pre-commit (80) - 12 new tests, 1240 total passing
- pysus ftp list: list all FTP datasets with descriptions - pysus ftp search: fuzzy search by name/description - pysus ftp show: display dataset details (groups, paths) - pysus ftp files: list available files with filters - pysus ftp download: download files with filters - 7 new CLI tests, 1247 total passing
- pysus dadosgov list/search/show/download commands - Token validation with helpful error messages - 7 new CLI tests, 1254 total passing
- pysus ducklake list/search/show/download commands - Mocked tests for offline testing - 5 new CLI tests, 1259 total passing
- pysus search: cross-origin dataset search with fuzzy suggestions - pysus info: show all available datasets - pysus open: web interface alias - 5 new CLI tests, 1264 total passing
…DATASUS databases - pysus/api/metadata/columns.py: load_column_metadata(), available_databases(), available_groups() - YAML schemas for SIH (rd/rj/er), SIA (pa/bi), SIM (do), SINASC (dn) - SINAN typecast fallback for columns not in YAML - 13 tests in pysus/tests/api/test_metadata_columns.py
- pysus/api/metadata/cache.py: get/set/invalidate/clear cache in ~/.cache/pysus/metadata/ - MD5-hashed cache keys, 7-day default TTL, JSON serialization - 10 tests covering roundtrip, expiry, invalidation, corruption handling
- pysus/api/metadata/versioning.py: get_schema_version(), list_schema_versions(), detect_schema_change(), schema_fingerprint() - Tracks year-to-schema mapping, detects added/removed/changed columns - 10 tests covering version lookup, change detection, fingerprinting
Exports: metadata (available_databases, load_column_metadata), quality (missing_values, validate_data, column_stats, quality_score, profile_report), transform (detect_units, link_datasets, aggregate_*, stream_parquet, mask_data, etc), export (to_csv, to_excel, to_geojson, to_sql), diff (diff_dfs, diff_summary, diff_rows)
Exports: cache_status, clear_cache, download_many, export, query_parquet, to_arrow
- Extract _params.py and _ui.py from _impl for shared param normalisation and CLI info/search helpers - Rewrite _impl/__init__.py with single 72-entry __all__ as canonical public API surface - Rewrite _impl/databases.py with origin routing (_fetch_data accepts origin param to route to specific clients) - Simplify pysus/__init__.py to single from pysus.api._impl import * - Add CLI saude.py sub-command group - Update CLI dadosgov.py, ftp.py, __init__.py to import from _impl._ui - Rewrite quality/profiling.py to remove flake8 E222/E221/E201/E202 - Fix cli/cache.py, cli/configure.py, api/types.py linting - Fix api/export/sql.py noqa for E702 - Update test_databases.py and test_adapters.py for client=None mocks
- Document all four data sources (FTP, DadosGov, Saude, DuckLake) - Add data quality, transformation, export, diff, column search sections - Document parallel downloads, streaming/DuckDB, cache management - Add full CLI command reference - Add TOML configuration and environment variable table - Expand data sources table with Saude-origin datasets - Document architecture with module tree - Add development setup and linting details
- Convert 28 SINAN metadata tarballs (1814 CSV rows) to 28 YAML schema files (1694 exported fields) in api/metadata/schemas/sinan/ - Skip non-exported fields (empty DBF, notes, multiline annotations) - Verified: 0 missing fields across all 28 diseases - All YAML files valid (pyyaml parse check) - Auto-discovered by existing column metadata loader - Remove 28 .tar.gz files from data/metadata/SINAN/ (typecast.py kept) - Fix pysus/__init__.py __all__ type annotation for mypy - Fix pysus/management/sync.py DuckLake None assert for mypy
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #321 +/- ##
=======================================
Coverage ? 95.13%
=======================================
Files ? 168
Lines ? 19444
Branches ? 0
=======================================
Hits ? 18498
Misses ? 946
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
… TOML path escaping
luabida
marked this pull request as ready for review
August 24, 2026 05:49
|
🎉 This PR is included in version 2.10.0 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.