ci(validate): gate Partner Node templates on openSource + API tag

[ryan-di]

PR Created by the Glary-Bot Agent

---

PR Created by the Glary-Bot Agent

Follow-up to #1124, which fixed three mislabelled Partner Node templates by hand. This adds the CI gate that catches the next one.

The gap

A template that calls a Partner Node is marked in two independent, hand-authored places, and nothing derives either one from the workflow JSON:

Marker	Consumer
"openSource": false	ComfyUI_frontend — isPartnerNode: template.openSource === false drives the subscription badge and the "Runs on" filter
the API tag	templates.comfy.org renders it as "Partner Nodes" (site/src/lib/tag-aliases.ts); scripts/mcp/sync_index.py turns it into the api MCP capability

So a Partner Node template can ship advertised as free. That is exactly what happened in #1124 — and two of those three were also missing the API tag, in a repo where 303 other templates had both.

The check

New check 8 in scripts/validate/validate_templates.py cross-references every workflow against the API node classes in scripts/data/mcp/api_node_ids.json (generated by scripts/mcp/scan_api_nodes.py from ComfyUI's comfy_api_nodes — living in that directory is what makes a node billable, so the list is complete by construction). It fails when:

1. a workflow calls a Partner Node but the entry is not "openSource": false;
2. a workflow calls a Partner Node but the entry has no API tag;
3. the two markers disagree with each other, in either direction;
4. a locale index disagrees with index.json on either marker.

Subgraph nodes are scanned too — in #1124 the Gemini nodes were nested inside definitions.subgraphs[], invisible to a top-level scan.

The reverse is deliberately not an error. Eight templates (api_ltx2_5_*, api_flux2, api_bfl_flux_1_kontext_*, …) legitimately carry both markers with no detectable node, because the committed node list lags a newly shipped vendor. Requiring a detectable node would redden main. Rule 3 still keeps them honest with each other.

The gate cannot no-op itself. An unusable api_node_ids.json is a hard error, not a silent pass. Verified by deleting the file in a simulated CI checkout:

❌ Validation failed with 1 error(s):
   • Cannot check Partner Node markers: api_node_ids.json not found at .../api_node_ids.json

Locale parity matters because sync-template-index.yml only auto-syncs same-repo PRs (if: github.event.pull_request.head.repo.full_name == github.repository) — a fork PR can leave 11 locale files behind, and non-English users read those files directly.

What it caught on main

One pre-existing violation, unrelated to #1124:

utility-gan_upscaler ("Fast GAN Video Upscaler") carried the API tag alongside "openSource": true. It contains no Partner Node at all — it is a local Real-ESRGAN workflow (UpscaleModelLoader, ImageUpscaleWithModel, LoadVideo, SaveVideo). Consequences of the stray tag:

• listed under Partner Nodes on templates.comfy.org;
• published to index.mcp.json with the api capability, so agents saw a local GAN upscaler as API-backed — its generated description even claims "API execution is supported for server-side upscaling".

Fixed by removing the tag (not by setting openSource: false — that would put a subscription badge on a genuinely free local workflow). Propagated to the 11 locale indexes with sync_data.py --index-only; the diff is exactly one line per index file. manifest.json is checksum-only, regenerated with sync_bundles.py as build-test.yml requires.

index.mcp.json is intentionally not regenerated here: sync_index.py also rewrites time-dependent freshness/recommend values for 16 unrelated templates. publish.yml and version-check.yml both regenerate it, so it self-heals on the next release.

Verification

Locally

• validate_templates.py → all 8 checks pass (2.1s; check 8 adds ~0.4s)
• validate_manifests.py → no errors (82 warnings, all pre-existing)
• pytest packages/core/tests → 106 passed. The 1 failure (test_sync_is_app.py::test_unreadable_permissions_are_reported) is environment-only — it chmod 000s a file and this sandbox runs as uid 0; it fails identically on unmodified main and will pass on a CI runner.
• ruff → my code adds 0 findings. validate_templates.py has the same 14 pre-existing errors as on main; the new test file is clean, which matters because build-test.yml lints packages/core.
• yaml-lint on the changed workflow → pass

26 new tests in packages/core/tests/test_partner_node_markers.py, run by build-test.yml on every PR. Confirmed non-vacuous by mutation: neutering find_partner_nodes to return [] fails 5 of them.

Simulated CI checkout. Copied only the paths in the workflow's sparse-checkout into a bare directory and ran the script there — passes, proving the added scripts/data/mcp/api_node_ids.json entry is sufficient. Then injected a new template calling a Partner Node inside a subgraph with openSource: true and no tag:

❌ Validation failed with 3 error(s):
   • index.json: 'new_partner_template' calls Partner Node(s) GeminiImage2Node but openSource is true
     - set "openSource": false so the template browser shows the subscription badge
   • index.json: 'new_partner_template' calls Partner Node(s) GeminiImage2Node but is missing the "API" tag
     - add it so the site and MCP index label it as a Partner Node template

In a real browser. Served this branch's index.json to a ComfyUI backend + frontend dev server. Fast GAN Video Upscaler still appears, now with only the Video Upscale chip and no badge — and searching "Upscaler" inside the Partner Nodes category returns the 22 real partner upscalers (Topaz, Magnific, WaveSpeed, HitPaw, Recraft) with the GAN upscaler correctly absent.

Screenshots below also cover the post-merge re-verification of #1124 on main that was requested alongside this: all three templates render the crown badge from main's data, and reverting just openSource to its pre-#1124 value makes the badge disappear on all three — while the API tags stay present, confirming the badge is driven solely by openSource: false.

Reviewer note

Removing the API tag from utility-gan_upscaler changes what appears under "Partner Nodes" on templates.comfy.org. That is the intended correction, but it is a user-visible data change, so please sanity-check that call.

Screenshots

[github-actions]

ComfyUI Node Compatibility Report

Informational only — this check does not block merging.

Baseline: static scan of ComfyUI source — reports removed/deprecated core nodes only.

Status: review suggested — some templates may use removed, deprecated, or outdated node settings.

Metric	Count
Checked workflows	581
Clean	487
With findings	94
Total findings	341 (0 error(s), 341 warning(s))

Baseline: mode static, branch master, commit 0696f61dced6, 836 node specs

At a glance

Severity	Templates	Findings
🟡 Warnings	94	341

Issue types

Type	Count
Deprecated node	341

What each issue type means

Type	Guidance
Deprecated node	Node is marked deprecated — consider migrating to the recommended replacement.

Details

🟡 Warnings — 94 template(s), 341 finding(s). Deprecated nodes — migrate when convenient

Template	Findings
3d_moge_panorama_to_mesh.json	1
3d_moge_perspective_to_mesh.json	1
api_bytedance_seedream4.json	1
api_bytedance_seedream_5_0_lite_t2i.json	1
api_elevenlabs_speech_to_speech.json	2
api_elevenlabs_text_to_dialogue.json	2
api_elevenlabs_text_to_sound_effects.json	2
api_elevenlabs_text_to_speech.json	2
api_elevenlabs_voice_isolation.json	1
api_grok_image_edit.json	1
api_ideogram_v4_t2i.json	1
api_openai_image_1_i2i.json	1
api_openai_image_1_inpaint.json	1
api_openai_image_1_t2i.json	1
api_seedance2_0_mini_r2v.json	1
api_seedance2_0_r2v.json	1
api_seedance2_0_r2v_4k.json	1
api_seedance2_0_r2v_real_human.json	1
api_seedance2_5_r2v.json	1
api_seedance2_5_video_editing.json	1
api_topaz_image_enhance.json	1
audio-chatterbox_tts.json	1
audio-chatterbox_tts_dialog.json	1
audio-chatterbox_tts_multilingual.json	1
audio-chatterbox_vc.json	1
audio_ace_step1_5_xl_base.json	1
audio_ace_step1_5_xl_sft.json	1
audio_ace_step1_5_xl_turbo.json	1
audio_ace_step_1_5_checkpoint.json	1
audio_ace_step_1_5_split.json	1
audio_ace_step_1_5_split_4b.json	1
audio_ace_step_1_5_split_llm.json	2
audio_ace_step_1_m2m_editing.json	1
audio_ace_step_1_t2a_instrumentals.json	1
audio_ace_step_1_t2a_song.json	1
audio_melbandroformer_audio_separation.json	2
audio_stable_audio_3_medium.json	1
audio_stable_audio_3_medium_base.json	1
audio_stable_audio_example.json	1
template_contact_sheet-step_3.app.json	7
template_eric_thumbnail_generator.json	1
template_graphic_color_remixer.json	29
template_image_speech_to_video.json	3
template_product_placement.json	1
template_purz_nb2_single_image_sprite_sheet.json	1
template_seedance2_0_viral_videos_character_swap.json	2
template_seedance2_storyboard_to_video.json	3
template_seedance_2_0_plus_llm_prompt_helper.json	2
template_sirolim_any_aspect_ratio_nb2.json	1
template_sirolim_image_script_video.json	2
template_sirolim_seamless_loop.json	1
templates-1_input-multiple_styles_prompt.app.json	12
templates-2x2_grid-character_bg_product.json	4
templates-2x2_grid-iso_miniatures.json	4
templates-3D_logo_texture_animation.json	1
templates-3x3_grid_brand_icons.json	9
templates-8x8_grid-pfp.json	65
templates-9grid_social_media-v2.0.json	10
templates-all_in_one-image_edit_models.json	3
templates-car_product.json	7
templates-fashion_shoot_prompt_doodle.json	5
templates-fashion_shoot_vton.json	6
templates-multiple_consistent_shots-nb_pro.json	1
templates-photo_to_product_vid.json	3
templates-poster_product_integration.json	3
templates-poster_to_2x2_mockups-v2.0.json	5
templates-product_ad-v2.0.json	1
templates-product_scene_relight.json	1
templates-split_stack.json	5
templates-sprite_sheet.json	32
templates-stitched_vid_contact_sheet.json	21
templates-textured_logotype-v2.1.json	1
templates_all_in_one_image_edit_models.app.json	4
templates_mjm_airt_machine_api.json	6
templates_mjm_image_to_3d.json	3
templates_mjm_looped_restyler.json	3
templates_product_scene_transformation.json	4
templates_purz_wan22_animate_auto_full_scene.json	1
templates_rob_fashion_shoot_vton-4in1.app.json	4
templates_rob_kling3_0_multishot_llm_product.json	1
utility-audioseparation.json	4
utility-topaz_landscape_upscaler.json	1
utility_moge_depth_estimation.json	1
utility_sirolim_image_controlled_upscale.json	1
utility_topaz_illustration_upscale.json	1
video_ltx2_3_i2v.json	1
video_ltx2_3_ia2v.json	1
video_ltx2_3_ic_lora.json	1
video_ltx2_3_id_lora.json	1
video_ltx2_3_t2v.json	1
video_ltx2_i2v.json	1
video_ltx2_i2v_distilled.json	1
video_ltx2_i2v_lora.json	1
video_wan2_2_14B_animate.json	2

---

This comment is automatically updated by the ComfyUI Node Compatibility Report workflow.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 19fd09ef-4cca-427e-a340-a4ed0e8ee7c4

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

🔤 Spellcheck Results

Check	Status
templates/index.json (titles & descriptions)	✅ No issues found
Workflow JSON notes (MarkdownNote / Note nodes)	✅ No issues found

All spellchecks passed! 🎉

[glary-bot]

@coderabbitai review

---

Re: the "Review skipped — Draft detected" status above — that is an automated status message, not feedback on the diff, so there is nothing to address from it. Triggering a real pass on the draft with the command above.

For context while you look, the diff is three things:

1. scripts/validate/validate_templates.py — new check 8, check_partner_node_markers. Cross-references every workflow (including definitions.subgraphs[].nodes) against the API node classes in scripts/data/mcp/api_node_ids.json and requires both Partner Node markers — "openSource": false and the API tag — on any template that calls one, plus agreement between the two markers in either direction, plus locale-index parity with index.json.

2. packages/core/tests/test_partner_node_markers.py — 26 tests, run by build-test.yml on every PR.

3. One data fix: utility-gan_upscaler had the API tag with "openSource": true and contains no Partner Node at all (local Real-ESRGAN). The tag is removed and propagated to the 11 locale indexes via sync_data.py --index-only; manifest.json is checksum-only.

Two design decisions worth reviewing specifically, both deliberate:

• The reverse direction is not an error. A template carrying both markers with no detectable Partner Node passes. Eight templates are in that state today (api_ltx2_5_*, api_flux2, api_bfl_flux_1_kontext_*) because the committed node list lags a newly shipped vendor. Requiring a detectable node would redden main.
• The API tag is compared literally in locale files. That is safe because scripts/data/i18n.json pins API to "API" in all 11 locales; verified empirically across all 581 templates × 12 index files (0 diffs).

One finding from the earlier review pass, already fixed in this commit: load_partner_node_classes originally filtered node_ids entries with isinstance(n, str). A bare string is iterable, so {"node_ids": "GeminiImage2Node"} loaded a set of single characters, matched no node type, and silently turned the whole gate into a no-op. It now rejects any payload that is not a non-empty list of non-empty strings, and cross-checks node_count. Tests cover string, scalar, object, array-root, mixed-type, blank-entry, and count-mismatch payloads.

CI status so far on d759b0d4: validate (the workflow this PR modifies) is green, and sync-index ran without needing to push a locale-sync commit.

[glary-bot]

Update: build-and-test was red on the first commit, now fixed in cb96a62f. All 16 checks are green.

This one is worth reading even if you skim the rest, because the failure had nothing to do with the new check's logic and everything to do with a latent trap in this script.

What happened. build-and-test failed with exit code 3 — pytest's internal error code, with zero tests run:

INTERNALERROR>     sys.exit(1)
INTERNALERROR> SystemExit: 1
mainloop: caught unexpected SystemExit!

Root cause. validate_templates.py called sys.exit(1) at import time when jsonschema was missing. That was harmless while the module was only ever run as a CLI. This PR adds packages/core/tests/test_partner_node_markers.py, which imports it — and build-test.yml installs only build pytest ruff aiohttp twine, no jsonschema. So importing the test module killed the interpreter mid-collection.

It passed locally purely because my sandbox had jsonschema installed from running the validator by hand.

Fix. jsonschema is set to None on ImportError and reported from main() instead. Nothing about the CLI contract changes:

$ python scripts/validate/validate_templates.py     # in a venv without jsonschema
Error: jsonschema package not installed. Run: pip install jsonschema
$ echo $?
1

Same message, same exit code, no traceback. The Validate Templates workflow installs jsonschema explicitly, so the schema check is untouched — and it is green on both commits. validate_schema() also returns that error rather than dereferencing None, in case it is ever called directly.

I deliberately did not add jsonschema to build-test.yml's install list. No test needs it, and a module that hard-exits the interpreter on import is worth fixing on its own — the next test file to import this script would have hit the same wall.

Verified against CI's exact environment, a fresh venv built from build-test.yml's install list (no jsonschema, pytest 9.1.1, ruff 0.16.3):

	before cb96a62f	after
pytest packages/core/tests	exit 3, 0 tests run	106 passed, exit 1
ruff check … packages/core packages/meta	—	clean

The remaining exit-1 failure is test_sync_is_app.py::test_unreadable_permissions_are_reported, which chmod 000s a file and asserts it cannot be read. My sandbox runs as uid 0, where that assertion cannot hold; it fails identically on unmodified main and passes on CI's non-root runner — confirmed by build-and-test now being green.

Also worth noting for the earlier concern about auto-sync races: sync-index ran on both commits and pushed nothing, because the locale indexes were already in sync from sync_data.py --index-only. No repeat of the drift that hit #1124.