Update make_fossa_deps_conan.py with smarter dependency classification - #1707
Update make_fossa_deps_conan.py with smarter dependency classification#1707jtrinidad-fossa wants to merge 3 commits into
Conversation
Merges improvements from an updated version of the script: - Replaces vendored-dependencies output with three-tier classification: referenced-dependencies (git), remote-dependencies (archive tarball), and custom-dependencies (fallback), matching fossa-deps.yml spec more closely - Adds git URL extraction from conandata.sources before falling back to homepage, supporting GitHub, GitLab, Bitbucket, SourceHut, and Codeberg - Adds git tag extraction from source URLs to handle packages that use non-standard tag formats (e.g. curl-8_17_0 vs 8.17.0) - Adds archive URL fallback for packages like GNU libraries that have no Git repo - Skips private-channel packages without conandata to avoid resolving internal version strings as git tags - Skips header-only libraries with binary == "Skip" - Adds Option B usage: accept a pre-generated conan graph info JSON file as an argument instead of always running conan via subprocess - Removes package_id from version string (was opaque to FOSSA's git resolver) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
WalkthroughThis pull request refactors a Python script that processes Conan v2 dependency graphs and generates FOSSA dependency manifests. The dataclass-based YAML model was replaced with five focused extraction helpers that derive normalized Git URLs, archive URLs, versions, and embedded tags from Conan node metadata. The core generator now filters ineligible nodes (root, build-context, test, header-only), classifies remaining dependencies as Git-referenced, remote-archived, or custom, and writes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Full details: Description checkExplanation The pull request description includes all required sections and provides a clear overview, acceptance criteria, concrete testing steps, risks, metrics, references, and checklist status. Minor wording issues and unchecked documentation/changelog items do not make the description incomplete. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/walkthroughs/make_fossa_deps_conan.py`:
- Around line 76-81: The current normalization truncates multi-segment GitLab
project paths by taking only the first two path parts; update the logic around
clean_url/host_name/parts/path_parts/git_url to preserve the full project path
for gitlab.com by joining all remaining path_parts (git_url = f"{host_name}/" +
"/".join(path_parts)), while still keeping the existing behavior of using only
owner/repo for other hosts (and preserving the return tuple (host_name ==
"github.com")). Apply the same change to the duplicate block referenced at the
other location (lines 108-111).
- Around line 167-169: The function extract_git_tag_from_source_url currently
strips a leading "v" from the captured archive tag (m.group(1).lstrip("v")),
which loses the exact git ref for repos that publish tags like "v1.2.3"; update
the function to return the raw captured tag (m.group(1)) without removing a
leading "v" so the exact archive tag is preserved.
- Around line 346-350: The current branch silently falls back to
get_graph_from_conan when a single JSON filename is passed but the file does not
exist; change the logic so that if len(args) == 1 and args[0].endswith(".json")
but os.path.isfile(args[0]) is False, the script fails fast with a clear error
(e.g., print/log an error and exit or raise SystemExit) rather than calling
get_graph_from_conan; update the conditional around
get_graph_from_file/get_graph_from_conan and use args, os.path.isfile,
get_graph_from_file, and get_graph_from_conan to implement the check and exit
behavior.
- Around line 251-293: The current emitter appends raw interpolated strings into
yaml_lines (see yaml_lines, the git_deps/archive_deps/custom_deps loops and the
final f.write) which can produce invalid YAML for values containing colons,
hashes, quotes or newlines; instead build proper Python data structures for
referenced-dependencies/remote-dependencies/custom-dependencies from
git_deps/archive_deps/custom_deps and serialize them with a YAML library (e.g.,
yaml.safe_dump(..., sort_keys=False) from PyYAML) when writing fossa-deps.yml so
all scalars are correctly quoted/escaped and the output is valid YAML.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro
Run ID: 7b94c08b-bd0b-4e72-9d3e-e1b6af9b101e
📒 Files selected for processing (1)
docs/walkthroughs/make_fossa_deps_conan.py
| parts = clean_url.split(host_name + "/") | ||
| if len(parts) > 1: | ||
| path_parts = parts[1].split("/") | ||
| if len(path_parts) >= 2: | ||
| git_url = f"{host_name}/{path_parts[0]}/{path_parts[1]}" | ||
| return git_url, (host_name == "github.com") |
There was a problem hiding this comment.
Preserve the full GitLab project path when normalizing forge URLs.
Both helpers currently assume every supported forge uses exactly owner/repo. That truncates valid GitLab subgroup repos like gitlab.com/group/subgroup/project down to gitlab.com/group/subgroup, so FOSSA will resolve the wrong repository.
Suggested fix
- if len(path_parts) >= 2:
- git_url = f"{host_name}/{path_parts[0]}/{path_parts[1]}"
+ if host_name == "gitlab.com":
+ repo_path = re.split(r"/-/|/releases/|/archive/", parts[1], maxsplit=1)[0].rstrip("/")
+ if repo_path:
+ git_url = f"{host_name}/{repo_path}"
+ return git_url, False
+ elif len(path_parts) >= 2:
+ git_url = f"{host_name}/{path_parts[0]}/{path_parts[1]}"
return git_url, (host_name == "github.com")Also applies to: 108-111
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/walkthroughs/make_fossa_deps_conan.py` around lines 76 - 81, The current
normalization truncates multi-segment GitLab project paths by taking only the
first two path parts; update the logic around
clean_url/host_name/parts/path_parts/git_url to preserve the full project path
for gitlab.com by joining all remaining path_parts (git_url = f"{host_name}/" +
"/".join(path_parts)), while still keeping the existing behavior of using only
owner/repo for other hosts (and preserving the return tuple (host_name ==
"github.com")). Apply the same change to the duplicate block referenced at the
other location (lines 108-111).
| m = re.search(r'/archive/([^/]+)\.(?:tar\.gz|tar\.xz|tar\.bz2|zip)$', url) | ||
| if m: | ||
| return m.group(1).lstrip("v") |
There was a problem hiding this comment.
Return the exact archive tag instead of stripping v.
extract_git_tag_from_source_url() is supposed to recover the real git ref. For URLs like /archive/v1.2.3.tar.gz, converting that to 1.2.3 will miss the tag on repos that actually publish v1.2.3.
Suggested fix
- return m.group(1).lstrip("v")
+ return m.group(1)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| m = re.search(r'/archive/([^/]+)\.(?:tar\.gz|tar\.xz|tar\.bz2|zip)$', url) | |
| if m: | |
| return m.group(1).lstrip("v") | |
| m = re.search(r'/archive/([^/]+)\.(?:tar\.gz|tar\.xz|tar\.bz2|zip)$', url) | |
| if m: | |
| return m.group(1) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/walkthroughs/make_fossa_deps_conan.py` around lines 167 - 169, The
function extract_git_tag_from_source_url currently strips a leading "v" from the
captured archive tag (m.group(1).lstrip("v")), which loses the exact git ref for
repos that publish tags like "v1.2.3"; update the function to return the raw
captured tag (m.group(1)) without removing a leading "v" so the exact archive
tag is preserved.
| # If the sole argument is a JSON file, read it directly; otherwise run conan. | ||
| if len(args) == 1 and args[0].endswith(".json") and os.path.isfile(args[0]): | ||
| data = get_graph_from_file(args[0]) | ||
| else: | ||
| data = get_graph_from_conan(args) |
There was a problem hiding this comment.
Fail fast when the requested JSON graph file does not exist.
If the user passes a single .json path with a typo, the script silently switches to Option A and invokes Conan instead. That makes the documented file-input mode very confusing to debug.
Suggested fix
- if len(args) == 1 and args[0].endswith(".json") and os.path.isfile(args[0]):
- data = get_graph_from_file(args[0])
+ if len(args) == 1 and args[0].endswith(".json"):
+ if not os.path.isfile(args[0]):
+ logging.error("JSON graph file not found: %s", args[0])
+ sys.exit(1)
+ data = get_graph_from_file(args[0])
else:
data = get_graph_from_conan(args)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| # If the sole argument is a JSON file, read it directly; otherwise run conan. | |
| if len(args) == 1 and args[0].endswith(".json") and os.path.isfile(args[0]): | |
| data = get_graph_from_file(args[0]) | |
| else: | |
| data = get_graph_from_conan(args) | |
| # If the sole argument is a JSON file, read it directly; otherwise run conan. | |
| if len(args) == 1 and args[0].endswith(".json"): | |
| if not os.path.isfile(args[0]): | |
| logging.error("JSON graph file not found: %s", args[0]) | |
| sys.exit(1) | |
| data = get_graph_from_file(args[0]) | |
| else: | |
| data = get_graph_from_conan(args) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/walkthroughs/make_fossa_deps_conan.py` around lines 346 - 350, The
current branch silently falls back to get_graph_from_conan when a single JSON
filename is passed but the file does not exist; change the logic so that if
len(args) == 1 and args[0].endswith(".json") but os.path.isfile(args[0]) is
False, the script fails fast with a clear error (e.g., print/log an error and
exit or raise SystemExit) rather than calling get_graph_from_conan; update the
conditional around get_graph_from_file/get_graph_from_conan and use args,
os.path.isfile, get_graph_from_file, and get_graph_from_conan to implement the
check and exit behavior.
# Conflicts: # docs/walkthroughs/make_fossa_deps_conan.py
- Preserve full GitLab group/subgroup path instead of truncating to owner/repo - Keep the 'v' prefix on archive-derived git tags instead of stripping it - Quote YAML scalars (name/version/license/url/homepage/description) via json.dumps so values with colons, '#', or quotes can't break the output - Fail fast with a clear error when a passed .json path doesn't exist, instead of silently falling back to running conan Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QGAeN1v9hkGHyXPtiPJSve
1 participant
Overview
Merges improvements from an updated version of the script:
Acceptance criteria
Customer's will be able to use this updated script as a starting point for scanning their Conan projects. The current hadn't been updated in a while and needed an update.
Testing plan
conan graph info -f jsonoutput supplied by a customer and ran it through the script (python3 make_fossa_deps_conan.py customer-graph.json) to confirm it produces a validfossa-deps.yml(Option B / pre-generated JSON path).{ "graph": { "nodes": { "0": {"label": "conanfile.txt", "recipe": "Consumer"}, "1": { "name": "zlib", "ref": "zlib/1.3.1", "version": "1.3.1", "license": ["Zlib", "MIT"], "homepage": "https://github.com/madler/zlib", "conandata": {"sources": {"1.3.1": {"url": ["https://github.com/madler/zlib/releases/download/v1.3.1/zlib-1.3.1.tar.gz"]}}} }, "2": {"name": "noconan", "ref": "noconan/2.0", "version": "2.0", "license": null} } } }python3 make_fossa_deps_conan.py fake-graph.jsonand inspect the generatedfossa-deps.yml: node1should land underreferenced-dependencieswith the extracted GitHub URL and tag, and node2(no source URL) should land undercustom-dependencieswithlicense: NOASSERTION(confirms the list/null license fallback works and doesn't emit invalid YAML).python3 -m py_compile make_fossa_deps_conan.pyfor a basic syntax/import check — this script lives underdocs/walkthroughs/and isn't wired into the Haskell/Rust unit or integration test suites, so manual execution against sample graphs is the only current verification path.fossa analyzeaccepts the generatedfossa-deps.ymlwithout schema errors (run againstdocs/references/files/fossa-deps.schema.jsonor viafossa analyzeon a scratch project).Risks
Would like someone more familiar with this script to make sure it is correct.
Metrics
Not applicable — this is a standalone script distributed via
docs/walkthroughs/, not compiled into the fossa-cli binary, so there's no telemetry/analytics hook to instrument. Adoption/success would show up indirectly as fewer Zendesk tickets about broken or stale Conan integration guidance.References
No tracked Jira/Zendesk ticket — this rewrite was driven directly by customer feedback (a customer-supplied
conan graph info.jsonsurfaced gaps in the previous version's git-URL/tag resolution and vendored-dependency classification).Checklist
docs/.docs/README.msand gave consideration to how discoverable or not my documentation is.Changelog.md. If this PR did not mark a release, I added my changes into an## Unreleasedsection at the top..fossa.ymlorfossa-deps.{json.yml}, I updateddocs/references/files/*.schema.jsonAND I have updated example files used byfossa initcommand. You may also need to update these if you have added/removed new dependency type (e.g.pip) or analysis target type (e.g.poetry).docs/references/subcommands/<subcommand>.md.