fix: proposal dedup misses cross-category and boilerplate-diluted duplicates - #544
Open
essentialbit wants to merge 3 commits into
Open
fix: proposal dedup misses cross-category and boilerplate-diluted duplicates#544essentialbit wants to merge 3 commits into
essentialbit wants to merge 3 commits into
Conversation
…luted duplicates _find_similar_proposal() hard-filtered candidates by exact category match before computing Jaccard overlap, and the category taxonomy has drifted over time (free-text like "macro-signal" vs later structured "l2_pattern_intelligence" slugs) -- so two proposals for the same idea filed months apart under different category conventions were never even compared. On real data (not the GitHub-rendered issue body) the pure word-overlap on the actual duplicate pair was 0.274, just under the 0.3 threshold, while an unrelated proposal scored 0.307 against the same target from shared macro-badge template boilerplate alone. Root-caused live: issue #543 (2026-08-01) duplicated issue #484 (2026-07-21), both proposing a FRED WEI weekly-economic-index badge. Fix: drop the category filter (still accepted for API stability, no longer used to gate candidates), and add a precise shared-FRED-series-ID check as a strong same-idea signal independent of the Jaccard threshold, since this codebase's proposals consistently name their data source as "FRED series X". Verified against a scratch copy of the live DB (never touched directly): the real duplicate now correctly dedupes, and a genuinely distinct FRED-series proposal still gets its own new id. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
memory_store._find_similar_proposal()hard-filtered dedup candidates by exactcategorymatch, but the category taxonomy has drifted over the project's life (free-textmacro-signalvs later structuredl2_pattern_intelligenceslugs) — so two proposals for the same idea filed under different category eras never got compared at all.Fix
Test plan
sentinel.db(never touched directly): with proposal feat: per-story HQ-to-exchange geocoding for globe arcs (Phase 1 of #48, sensor-implemented) #52 deleted and re-submitted with its original title/description,insert_feature_proposalnow correctly returns id 28 (the pre-existing duplicate) instead of creating a new row.python3 -c "from main import *"still imports cleanly.tests/absent) — manual scratch-DB verification only.Note: issues #484 and #543 (the real duplicate pair that surfaced this bug) are left open for the user to consolidate/close — sensor policy is to flag duplicates, not bulk-close them unilaterally.
🤖 Generated with headless FredAI sensor cycle