fix: apply similarity threshold on raw cosine before normalization

README.md

@@ -80,7 +80,7 @@ Available in the Configuration screen's **Note Graph** section:
 | Setting | Default | Effect |
 |---|---|---|
 | Enable AI-based semantic analysis | Off | Adds semantic similarity edges using Joplin AI |
-| Similarity threshold | 50% | Lower values surface more semantic edges |
+| Similarity threshold | 70% | Lower values surface more semantic edges |
 | Max semantic edges per note | 5 | Caps how many semantic connections each note keeps |
 | Enable LLM analysis | Off | Adds Pass B category labels and relationship explanations |
 | Retry AI embedding | Off | One-shot: re-runs AI-based semantic analysis, reusing cached embeddings |

docs/settings.md

@@ -10,7 +10,7 @@ normal.
 | Setting | Key | Type | Default | Effect |
 |---|---|---|---|---|
 | Enable AI-based semantic analysis | `noteGraph.aiAnalysisEnabled` | Boolean | `false` | Turns semantic edges on or off. Requires Joplin AI to be enabled with a ready embedding index (Configuration screen's AI page). |
-| Similarity threshold (%) | `noteGraph.similarityThreshold` | Integer, 0-100, step 5 | `50` | Minimum bonus-boosted similarity score for a semantic edge to appear, as a percentage. Lower = more edges. Only applies when AI analysis is enabled. |
+| Similarity threshold (%) | `noteGraph.similarityThreshold` | Integer, 0-100, step 5 | `70` | Minimum content similarity (raw cosine) for a semantic edge to appear, as a percentage. Lower = more edges. Only applies when AI analysis is enabled. |
 | Max semantic edges per note (top-K) | `noteGraph.maxEdgesPerNote` | Integer, 1-20, step 1 | `5` | Caps how many of each note's strongest semantic connections are kept. Only applies when AI analysis is enabled. |
 | Enable LLM analysis | `noteGraph.llmEnrichmentEnabled` | Boolean | `false` | Turns on Pass B: category labels and relationship explanations via Joplin AI chat. Requires AI-based semantic analysis to also be enabled. See [LLM enrichment](llm-enrichment.md). |
 | Retry AI embedding | `noteGraph.retryEmbedding` | Boolean | `false` | One-shot trigger, not a persistent toggle: ticking it immediately retries AI-based semantic analysis (for example, after cancelling it), then unticks itself. No-op if the graph panel hasn't been opened yet. |

docs/similarity-engine.md

@@ -61,9 +61,16 @@ miss/no-op rather than failing the embed, so a corrupt cache degrades to
 fixed pipeline over every candidate note pair:
 
 ```
-raw scores -> floor -> normalize -> add bonuses -> threshold -> top-K
+raw scores -> floor -> threshold (raw) -> normalize -> bonuses -> top-K
 ```
 
+The **threshold is checked on the raw score, before normalization**. Min-max
+normalization always maps the batch's most-similar pair to exactly 1.0, so a
+post-normalization threshold could never reject it — even in a vault of
+completely unrelated notes, the closest pair would be scaled up and pass.
+Checking the raw cosine first gives the threshold an absolute meaning;
+normalization then only ranks the pairs that already passed.
+
 ### 1. Raw scores
 
 - **Vaults of 300 notes or fewer** (`LARGE_VAULT_THRESHOLD`): plain O(n²)
@@ -82,19 +89,27 @@ raw scores -> floor -> normalize -> add bonuses -> threshold -> top-K
 Pairs scoring below `SEMANTIC_FLOOR` (0.3) on the **raw** scale are dropped,
 unless the two notes are already directly linked (those are kept and
 resolved later, at the threshold step). This has to happen before
-normalization: min-max normalization always stretches the best pair in the
-batch to exactly 1.0, even in a vault of totally unrelated notes, so a floor
-applied *after* normalization could never reject anything. Flooring the raw
-score is what gives 0.3 an absolute, not batch-relative, meaning.
+normalization: min-max normalization would always stretch the best pair in
+the batch to exactly 1.0, even in a vault of totally unrelated notes, so a
+floor applied *after* normalization could never reject anything. Flooring the
+raw score is what gives 0.3 an absolute, not batch-relative, meaning.
+
+### 3. Threshold
 
-### 3. Normalize
+Pairs whose raw score is below the configured threshold
+(`DEFAULT_THRESHOLD` = 0.7, user-adjustable) are dropped. Like the floor,
+this runs on the raw scale, *before* normalization, so the threshold is an
+absolute similarity bar rather than a batch-relative one — an unrelated pair
+that happens to be a batch's closest cannot be normalized up to pass.
+Normalization then only ranks the pairs that already cleared it.
 
-Surviving scores, including any sub-floor pairs kept for being directly
-linked, are min-max normalized to `[0, 1]` together. If the spread between
-the batch's min and max is under 0.1, normalization is skipped (there is
-nothing meaningful to stretch).
+### 4. Normalize
 
-### 4. Bonuses
+Surviving scores are min-max normalized to `[0, 1]` together. If the spread
+between the batch's min and max is under 0.1, normalization is skipped (there
+is nothing meaningful to stretch).
+
+### 5. Bonuses
 
 Three additive bonuses nudge the normalized score:
 
@@ -109,10 +124,8 @@ Three additive bonuses nudge the normalized score:
 They are excluded from the tag-overlap bonus because sharing them says
 nothing about content similarity.
 
-### 5. Threshold
-
-Pairs whose bonus-boosted score is below the configured threshold
-(`DEFAULT_THRESHOLD` = 0.5, user-adjustable) are dropped.
+Because the threshold and floor are checked on the *raw* score, bonuses can
+rank pairs but can never manufacture an edge out of a weak semantic score.
 
 ### 6. Top-K
 

docs/troubleshooting.md

@@ -26,12 +26,12 @@ simply the expected result. See [Settings reference](settings.md).
 
 ## The graph has very few or no semantic edges
 
-- Check the **similarity threshold** setting; 50% is the default and can be
+- Check the **similarity threshold** setting; 70% is the default and can be
   lowered to surface more edges.
 - A small vault, or a vault with genuinely unrelated notes, will produce
-  fewer edges by design: the raw-score floor (`SEMANTIC_FLOOR`, 0.3) exists
-  specifically to prevent tag or link bonuses alone from manufacturing an
-  edge out of a weak semantic score. See [Similarity
+  fewer edges by design: semantic edges require the raw cosine similarity to
+  clear the threshold *before* normalization, and no tag, link, or time bonus
+  can manufacture one out of a weak semantic score. See [Similarity
   engine](similarity-engine.md).
 - Confirm the embedding index state is `ready` or at least `indexing` with
   meaningful progress, not `preparing`.

src/manifest.json

@@ -2,7 +2,7 @@
 	"manifest_version": 1,
 	"id": "com.yugalkaushik.plugin-note-graph",
 	"app_min_version": "3.5",
-	"version": "1.0.0",
+	"version": "1.0.1",
 	"name": "Note Graph Plugin",
 	"description": "Visualizes your notes as an interactive graph, connecting them by links, tags and AI-detected semantic similarity.",
 	"author": "yugalkaushik",

src/services/__snapshots__/AnalysisController.golden.test.ts.snap

@@ -63,6 +63,24 @@ exports[`AnalysisController (full pipeline golden set) renders the enriched sema
         "type": "semantic",
       },
     },
+    {
+      "data": {
+        "id": "11111111111111111111111111111111::44444444444444444444444444444444::semantic",
+        "score": 0.15000000000000002,
+        "source": "11111111111111111111111111111111",
+        "target": "44444444444444444444444444444444",
+        "type": "semantic",
+      },
+    },
+    {
+      "data": {
+        "id": "22222222222222222222222222222222::44444444444444444444444444444444::semantic",
+        "score": 0.1,
+        "source": "22222222222222222222222222222222",
+        "target": "44444444444444444444444444444444",
+        "type": "semantic",
+      },
+    },
     {
       "data": {
         "id": "33333333333333333333333333333333::44444444444444444444444444444444::semantic",
@@ -79,7 +97,7 @@ exports[`AnalysisController (full pipeline golden set) renders the enriched sema
       "data": {
         "category": "Gardening",
         "community": 0,
-        "degree": 6,
+        "degree": 7,
         "id": "11111111111111111111111111111111",
         "label": "Alpha Note",
         "noteId": "11111111111111111111111111111111",
@@ -90,11 +108,11 @@ exports[`AnalysisController (full pipeline golden set) renders the enriched sema
       "data": {
         "category": "Gardening",
         "community": 0,
-        "degree": 4,
+        "degree": 5,
         "id": "22222222222222222222222222222222",
         "label": "Beta",
         "noteId": "22222222222222222222222222222222",
-        "size": 6,
+        "size": 7,
       },
     },
     {
@@ -112,11 +130,11 @@ exports[`AnalysisController (full pipeline golden set) renders the enriched sema
       "data": {
         "category": "Gardening",
         "community": 0,
-        "degree": 3,
+        "degree": 5,
         "id": "44444444444444444444444444444444",
         "label": "Delta",
         "noteId": "44444444444444444444444444444444",
-        "size": 1,
+        "size": 6,
       },
     },
   ],

src/services/settings/GraphSettings.test.ts

@@ -30,7 +30,7 @@ describe('GraphSettings', () => {
 					}),
 					'noteGraph.similarityThreshold': expect.objectContaining({
 						type: SettingItemType.Int,
-						value: 50,
+						value: 70,
 						minimum: 0,
 						maximum: 100,
 						public: true,
@@ -122,7 +122,7 @@ describe('GraphSettings', () => {
 
 			expect(result.threshold).not.toBeNaN();
 			expect(result.topK).not.toBeNaN();
-			expect(result).toEqual({ threshold: 0.5, topK: 5 });
+			expect(result).toEqual({ threshold: 0.7, topK: 5 });
 		});
 
 		it('clamps an out-of-range threshold and topK to the registered min/max', async () => {
@@ -144,7 +144,7 @@ describe('GraphSettings', () => {
 
 			const result = await getSimilaritySettings();
 
-			expect(result).toEqual({ threshold: 0.5, topK: 5 });
+			expect(result).toEqual({ threshold: 0.7, topK: 5 });
 		});
 
 		it('falls back to defaults instead of clamping to the minimum when a value is null, empty, or a boolean', async () => {
@@ -155,7 +155,7 @@ describe('GraphSettings', () => {
 
 			const result = await getSimilaritySettings();
 
-			expect(result).toEqual({ threshold: 0.5, topK: 5 });
+			expect(result).toEqual({ threshold: 0.7, topK: 5 });
 
 			(joplin.settings.values as jest.Mock).mockResolvedValue({
 				'noteGraph.similarityThreshold': false,
@@ -164,7 +164,7 @@ describe('GraphSettings', () => {
 
 			const secondResult = await getSimilaritySettings();
 
-			expect(secondResult).toEqual({ threshold: 0.5, topK: 5 });
+			expect(secondResult).toEqual({ threshold: 0.7, topK: 5 });
 		});
 	});
 

src/services/similarity/SimilarityEngine.test.ts

@@ -423,12 +423,14 @@ describe('SimilarityEngine', () => {
 
 	describe('top-K filtering', () => {
 		it('limits edges per note', async () => {
+			// 30° steps keep each note's two neighbours above the raw threshold
+			// (cos 30° ≈ 0.87), so top-K has something to limit.
 			const notes = [];
 			const embedded = [];
 			for (let i = 0; i < 6; i++) {
 				notes.push(makeNote(`n${i}`, `Note ${i}`));
 				embedded.push(
-					embed(`n${i}`, [Math.cos((i * Math.PI) / 3), Math.sin((i * Math.PI) / 3)])
+					embed(`n${i}`, [Math.cos((i * Math.PI) / 6), Math.sin((i * Math.PI) / 6)])
 				);
 			}
 
@@ -484,15 +486,15 @@ describe('SimilarityEngine', () => {
 		});
 
 		it('applies a looser caller-supplied threshold that admits a pair DEFAULT_THRESHOLD would reject', async () => {
-			// Raw cosine 0.35 (above SEMANTIC_FLOOR) plus the same-day temporal
-			// bonus (0.1) lands at 0.45 — below DEFAULT_THRESHOLD (0.5) but above
-			// a caller-supplied 0.4.
+			// Raw cosine 0.55 clears SEMANTIC_FLOOR but sits below
+			// DEFAULT_THRESHOLD (0.7); a caller-supplied 0.5 admits it. The
+			// threshold is checked on the raw score, before normalization.
 			const notes = [makeNote('a', 'A'), makeNote('b', 'B')];
-			const embedded = [embed('a', [1, 0]), embed('b', [0.35, Math.sqrt(1 - 0.35 * 0.35)])];
+			const embedded = [embed('a', [1, 0]), embed('b', [0.55, Math.sqrt(1 - 0.55 * 0.55)])];
 
 			const engine = new SimilarityEngine(notes, embedded);
 			const defaultPairs = await engine.compute();
-			const loosePairs = await engine.compute(0.4);
+			const loosePairs = await engine.compute(0.5);
 
 			expect(defaultPairs).toEqual([]);
 			expect(loosePairs).toHaveLength(1);
@@ -503,12 +505,13 @@ describe('SimilarityEngine', () => {
 			// keeps it in its own top-K), so topK=0 is the only value that
 			// unambiguously proves the override took effect: every note's own
 			// kept list is empty, so no pair can survive from any side.
+			// 30° steps keep adjacent pairs above the raw threshold.
 			const notes = [];
 			const embedded = [];
 			for (let i = 0; i < 6; i++) {
 				notes.push(makeNote(`n${i}`, `Note ${i}`));
 				embedded.push(
-					embed(`n${i}`, [Math.cos((i * Math.PI) / 3), Math.sin((i * Math.PI) / 3)])
+					embed(`n${i}`, [Math.cos((i * Math.PI) / 6), Math.sin((i * Math.PI) / 6)])
 				);
 			}
 
@@ -581,6 +584,35 @@ describe('SimilarityEngine', () => {
 		});
 	});
 
+	describe('threshold is applied to the raw score, before normalization', () => {
+		it('rejects a weak pair even when it is the closest pair in the batch', async () => {
+			const notes = [makeNote('a', 'A'), makeNote('b', 'B'), makeNote('c', 'C')];
+			const embedded = [
+				embed('a', [1, 0, 0]),
+				embed('b', [0.45, Math.sqrt(1 - 0.45 * 0.45), 0]),
+				embed('c', [0.32, 0, Math.sqrt(1 - 0.32 * 0.32)]),
+			];
+
+			const engine = new SimilarityEngine(notes, embedded);
+			const pairs = await engine.compute();
+
+			expect(pairs).toEqual([]);
+		});
+
+		it('cannot manufacture a semantic edge from a shared tag or direct link below the raw threshold', async () => {
+			const notes = [
+				makeNote('a', 'A', ['b'], ['shared']),
+				makeNote('b', 'B', [], ['shared']),
+			];
+			const embedded = [embed('a', [1, 0]), embed('b', [0.45, Math.sqrt(1 - 0.45 * 0.45)])];
+
+			const engine = new SimilarityEngine(notes, embedded);
+			const pairs = await engine.compute();
+
+			expect(pairs).toEqual([]);
+		});
+	});
+
 	describe('large vault (search-based) path retry', () => {
 		function makeLargeVault(): { notes: Note[]; embedded: EmbeddedNote[] } {
 			const count = LARGE_VAULT_THRESHOLD + 1;

src/services/similarity/SimilarityEngine.ts

@@ -45,14 +45,17 @@ export class SimilarityEngine {
 
 	/**
 	 * Orchestrates the full similarity pipeline:
-	 * compute → floor (raw scores) → normalize → enrich → threshold → top-K.
+	 * compute → floor (raw scores) → threshold (raw scores) → normalize →
+	 * bonuses → top-K.
 	 *
-	 * SEMANTIC_FLOOR is applied to *raw* scores, before normalization. Min-max
-	 * normalization always maps the batch's most-similar pair to exactly 1.0,
-	 * so a post-normalization floor can never reject it — even in a vault of
-	 * completely unrelated notes. Flooring on the raw scale (where 0.3 has an
-	 * absolute meaning) is what actually guarantees that tags alone can never
-	 * manufacture an edge out of a weak semantic score.
+	 * The threshold is checked on the *raw* score, before normalization.
+	 * Min-max normalization always maps the batch's most-similar pair to
+	 * exactly 1.0, so a post-normalization threshold could never reject it —
+	 * even in a vault of completely unrelated notes, the closest pair would be
+	 * scaled up and pass. Checking the raw cosine first gives the threshold an
+	 * absolute meaning; normalization then only ranks the pairs that already
+	 * passed. SEMANTIC_FLOOR (also raw, before normalization) guards against
+	 * tag-only edges on weak scores.
 	 */
 	public async compute(
 		threshold: number = DEFAULT_THRESHOLD,
@@ -75,10 +78,15 @@ export class SimilarityEngine {
 			return [];
 		}
 
-		const normalized = this.normalize(aboveFloor, SEMANTIC_FLOOR);
+		const aboveThreshold = this.filterBelowThreshold(aboveFloor, threshold);
+
+		if (aboveThreshold.length === 0) {
+			return [];
+		}
+
+		const normalized = this.normalize(aboveThreshold, SEMANTIC_FLOOR);
 		const enriched = this.addBonusPoints(normalized);
-		const aboveThreshold = this.filterBelowThreshold(enriched, threshold);
-		const topPairs = this.selectTopK(aboveThreshold, topK);
+		const topPairs = this.selectTopK(enriched, topK);
 
 		return topPairs;
 	}
@@ -122,8 +130,8 @@ export class SimilarityEngine {
 	 * property without invoking it (see JoplinNativeProvider.validateAiApi for why).
 	 *
 	 * Score-scale assumption: search relevance scores are treated as raw
-	 * similarity scores and flow through the same floor → normalize pipeline
-	 * as cosine scores.
+	 * similarity scores and flow through the same floor → threshold →
+	 * normalize pipeline as cosine scores.
 	 *
 	 * Failure handling: each note's search call is retried on transient
 	 * failures before being skipped; a partial candidate set is still
@@ -313,7 +321,7 @@ export class SimilarityEngine {
 		return pairs.filter((p) => p.score >= floor || this.isDirectlyLinked(p));
 	}
 
-	/** Keeps only pairs whose bonus-boosted score clears the threshold. */
+	/** Keeps only pairs whose raw score clears the threshold, before normalization. */
 	private filterBelowThreshold(pairs: SimilarityPair[], threshold: number): SimilarityPair[] {
 		return pairs.filter((p) => p.score >= threshold);
 	}

src/services/similarity/ThresholdPresets.ts

@@ -1,7 +1,7 @@
 /** Only a direct link bypasses this floor — tags alone can never create an edge below it. */
 export const SEMANTIC_FLOOR = 0.3;
 
-export const DEFAULT_THRESHOLD = 0.5;
+export const DEFAULT_THRESHOLD = 0.7;
 
 export const TOP_K = 5;