idea-spark: full validator suite passes when the candidate's core claim mischaracterizes prior work

[tianlrz]

Hi, and thanks for releasing this suite — I've been using idea-spark on a real direction for the past few days and it caught several bad ideas of mine cheaply, which is exactly what I hoped for.

I ran into one failure mode I couldn't find an existing gate for, and I wanted to report it in case it's useful. It's possible I'm misusing the skill, so please correct me if so.

What happened

Direction: credit assignment in agentic RL post-training under an outcome-only verifiable reward.

Across two run dirs I went through four candidate framings (two abandon, two reaching Phase 4). The final candidate rendered all three idea cards and passed the full validator suite:

7 pass, 0 warn, 0 fail

Its core claim was, in substance: "a family of step-credit maps (IAPO, GiGPO, VICT, SPA-RL, CVT-RL) shares mass conservation + sign preservation, and that safeguard costs them something measurable."

Afterwards I extracted the verbatim advantage equations from the five papers' method sections in phase0/fulltext_cache.json (one sub-agent call, ~1 hour, no GPU). The premise did not survive:

• SPA-RL — r_t^fused = α·c_t + β·g_t (Eq. 8), where g_t is an environment executability signal, and advantages come from GAE with a learned critic V_φ. It injects signal the terminal verifier never produced, so it is not a redistribution of the episode advantage at all.
• CVT-RL — learned value function, a learned outcome model, and extra frozen-policy rollouts. Same conclusion.
• GiGPO — states no conservation proposition and no sign property anywhere in intro+method. I also found counter-verdict emissions inside its own declared ranges (ω ∈ ℝ_{≥0}, γ ∈ (0,1]): e.g. R=[0,0,1,1], T=[2,3,4,5], γ=0.95, ω=2.0 gives A^E = +1.000 but A = −1.000.
• IAPO — genuinely locked, and algebraically so: w_i > 0 follows from Eq. (4)'s exp(...)/positive-normalizer, for all real β⁺, c, not just the published defaults.

So the "family" the candidate was built on has one member. The idea was not wrong in its internal logic — the gates that check that all passed — it was wrong about what the cited papers do.

Why I think the existing gates don't cover this

I went looking for the mechanism before writing this, and there is one: bottleneck_identify.txt §8b, the anchor_rule_pinning ledger, with the nice rule that a rule may only be listed as pinned if you can quote the sentence that fixes it. In my first run it fired correctly and reported sufficient_for_faithful_generator: false.

But as far as I can tell it doesn't reach this case, for three reasons:

1. It is anchor-scoped. The object is anchor_rule_pinning and exactly one closest_adjacent entry is the anchor. A family-level claim ranges over N methods. In my second run the anchor was the audit paper, not any of the methods the claim was actually about.
2. It pins free parameters, not class membership. The enumerated coverage is the selection predicate, population sizes, update rule, iteration count, tie-breaking — knob values. What broke me was predicate-shaped: does M conserve, and to what, does M use a learned critic, does M consume signal outside the terminal verifier. unpinned is even defined as "free parameter".
3. The deep-read pool isn't guaranteed to cover the set. The pool is chosen by a relevance heuristic before Phase 1 knows which papers matter. My run only had all five methods deep-read because I ran phase1_fulltext_topup by hand.

Downstream, paper_pointed_threat asks whether prior work subsumes my idea, which is the opposite direction from whether my characterization of prior work is accurate; the 2.3 gate checks the candidate's own procedure by execution; 4.1.5 checks implementability. None of them reads the cited papers' method sections back.

Suggestion

When the candidate's core claim is universally quantified over a set of prior methods, it would help to require a per-method grounding ledger — the same "quote the sentence or mark it unpinned" discipline as §8b, but (a) over every method the claim ranges over rather than the anchor alone, (b) covering class-membership predicates rather than only free parameters, and (c) with the fulltext pool forced to cover that set. A NOT_IN_FULLTEXT outcome is genuinely useful here: in my case VICT's row is honestly unfillable, because δ̂'s sign convention lives in Appendix D and λ, c are unstated in the body.

I'm happy to share the run directory, the validator output, and the extracted-formula JSON if any of that is useful.

Separately

During Phase 0 the Semantic Scholar connector returned 0 records for every query (rate limiting), but lit_grounding_mode still reported real and no degraded marker was written, so the 6–24-month published channel was silently missing from the corpus that the first two framings were built on. That looks like a distinct issue from the above — happy to file it separately if you'd like.

Thanks again for the work.