Feat/mandi publish tools - CLI - #39
Open
kelvinprabhu wants to merge 16 commits into
Open
kelvinprabhu wants to merge 16 commits into
kelvinprabhu wants to merge 16 commits into
Conversation
…data - Implemented main functionality to authenticate and collect market master data across India. - Added configuration handling for base URL, states, date range, and output file. - Introduced data collection logic with error handling for state-specific failures. - Created mappings for market-commodity and master data to facilitate data transformation. - Developed comprehensive tests to ensure functionality and error handling. - Embedded mappings into the binary to avoid external dependencies.
…ctionality - Introduced a new YAML mapping file for generating Beckn catalog payloads from Agmarknet market data. - Implemented HTTP client for publishing catalogs to the provider adapter. - Added tests for handling scenarios with no data from upstream and ensuring proper error reporting. - Created a run function to collect data and build catalogs without generating a collection document. - Enhanced publish functionality to handle transport errors and catalog retirement.
- Updated `build_test.go` to improve market resource validation and ensure accurate reporting of geometry-related issues. - Introduced `chunk_test.go` to validate chunking behavior for states exceeding geometry limits and ensure proper handling of markets without coordinates. - Enhanced `main.go` to improve catalog reporting, including detailed summaries of excluded markets and geometry-less markets. - Added `publish.go` changes to correctly associate catalog files with their respective states, accommodating split states. - Created `report_test.go` to validate the accuracy of reports generated for excluded markets and chunked states. - Updated `catalog.yaml` to streamline market object creation and ensure correct catalog IDs. - Removed redundant tests in `main_test.go` to maintain clarity and focus on essential functionality.
Move serveMappings() and newMapper() from mandi_publish/mappings.go to internal/catalogpublish/mapper.go as public functions that accept an embed.FS and directory name. Update all call sites in mandi_publish to use the new public API. This generalizes the mapper setup for reuse by other tools.
Const block was space-indented instead of tab-indented, with a stray blank line before the following const block. No functional change.
…nd improve build process
…coverage area to catalog mapping Bring the catalog/publish payload closer to the OpenAgriNet target schema: resourceAttributes.source, resource longDesc, provider shortDesc, and an ISO-3166-2 state coverageArea alongside the existing district one. Left market.location.geo out deliberately — coverageAreas is already the sole geometry copy, per the existing per-catalog geometry budget.
…ing, and main execution flow
|
📊 Test Coverage: ✅ Passed — 84% of changed lines covered, min 80% |
🛡️ Trivy security scan (CRITICAL,HIGH,MEDIUM,LOW)Go dependenciesNo findings at CRITICAL,HIGH,MEDIUM,LOW. Container image
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What's in this PR
This PR adds a new
mandi_publishCLI for publishing Agmarknet Vistaar market data, along with shared infrastructure for catalog publishing.1. Added
mandi_publishAdds a CLI under:
It:
catalog/publishpayloads for each state.Example:
Use
--publishto send the generated catalog to the adapter:2. Added shared
internal/catalogpublishtransportIntroduces:
This provides the shared transport used by
mandi_publishto sendcatalog/publishrequests.The transport is intended to be reusable by other publishing tools in the future.
3. Updated the adapter Docker image
Dockerfile.adapter-with-pluginsnow builds all tools under:As a result,
mandi_publishis available inside the adapter image at:4. Updated the Mandi catalog structure
The Mandi catalog was updated to follow the canonical OpenAgriNet conventions:
openagrinet.github.io/network-specs/...URLs instead of the staleraw.githubusercontent.com/...URLs.resourceAttributes.source.coverageAreasusing ISO-3166-2.longDescandshortDescfields expected by the schema.5. End-to-end verification
The complete flow was tested against a local OAN quick-start stack:
Verified successfully:
Running inside Docker
docker exec \ -e MANDI_TOKEN_USER=your_user \ -e MANDI_TOKEN_SECRET=your_secret \ -e MANDI_API_URI=http://34.0.4.235:8080 \ -e MANDI_PUBLISH_URL=http://localhost:9200 \ -e MANDI_PARTICIPANT_ID=agmarknet-live \ -e APP_NETWORK_ID=oan-dev \ onix-adapter \ /app/bin/mandi_publish \ --from 01-09-2026 \ --to 30-09-2026 \ --catalog-out /app/catalog \ --publish