[Detail Bug] Local docs ingestion: guard-rejected updated file leaves stale indexed vectors and citation snippets

[detail-app]

Detail Bug Report

https://app.detail.dev/org_befd6425-a158-4e24-9d4d-1e5c08769515/bugs/bug_6c5466fd-f615-4b0b-a50b-e3538f9a9ed5

Introduced in #114 by @WilliamAGH on Jul 18, 2026

Summary

• Context: LocalDocsFileIngestionProcessor owns the per-file local documentation ingestion lifecycle, including the "previously-indexed file changed and must be reconciled" handling that keeps the Qdrant index, the parsed-chunk store, and the file marker mutually consistent.
• Bug: When a previously-ingested local file's on-disk content changes such that HtmlContentGuard rejects it (e.g. the mirror re-fetches the URL and overwrites a previously-good file with a guard-rejectable HTTP 200 body — a Cloudflare "Just a moment…" challenge, or an "access denied"/"page not found" body whose title/heading HtmlContentGuard matches — see Trigger below), the quarantine path (prepare → quarantineRejectedFile) neither deletes the existing Qdrant points for the URL, nor prunes the now-orphaned local chunk state, nor advances the file marker. The old vectors stay indexed and citable under the same URL while the URL stays mirrored-and-rejected, while the file on disk no longer contains that content anywhere.
• Actual vs. expected: This must behave like processExcludedPage (the in-codebase rule for the same "previously-indexed file at this URL is no longer indexable" situation), which deletes the URL's vectors, prunes local state, and writes a fresh FileIngestionRecord with chunkHashes = List.of(). Instead the quarantine branch discards the already-computed requiresFullReindex (always true for a changed previously-ingested file) and the markerContext, returning a bare failedFile that touches only the quarantine copy.
• Impact: A retrieval returns a citation whose backing URL now serves rejected content (soft-404 / challenge / blank page), while the retrieved text is the old chunk text stored in the Qdrant point's doc_content payload (the retained prior-good points that failedFile never deletes). That stored text is communicated to the user through two paths (see §User-facing impact path): it is fed verbatim into the RAG prompt that generates the answer and surfaced as the citation snippet the user reads. Meanwhile the citation url is the canonical upstream HTTP URL that DocsSourceRegistry.resolveMirroredIngestionIdentities (DocsSourceRegistry.java:1061) projects from the local mirror file (via CitationRoute.resolveCitationUrl → canonicalizeHttpDocUrl, then stored in URL_FIELD payload by QdrantScoredPointDocumentMapper.applyKnownMetadata) — so the user follows the upstream link expecting the indexed text and gets the same Cloudflare challenge / soft-404 body that the mirror itself received (the upstream and the mirror file are the same resource class). The marker retains the old fingerprint, so the fingerprint mismatch re-fires on every subsequent run while the URL remains mirrored-and-rejected.

Code with Bug

// LocalDocsFileIngestionProcessor.java, prepare() — content-guard rejection branch:
            if (!excludedJavaApiPage && !excludedNavigationPage) {
                var contentGuard = fileContentServices.contentGuard();
                GuardDecision guardDecision = contentGuard.evaluate(new GuardInput(bodyText, parsedDocument));
                if (!guardDecision.acceptable()) {
                    String rejectionReason = guardDecision.rejectionReason();
                    return deferred(() -> quarantineRejectedFile(file, rejectionReason)); // <-- BUG 🔴 requiresFullReindex/markerContext discarded; no vector/marker/chunk cleanup
                }
            }

// quarantineRejectedFile — forensic copy only; never touches Qdrant, the parsed-chunk store, or the marker:
    private LocalDocsFileOutcome quarantineRejectedFile(Path file, String rejectionReason) {
        try {
            var quarantineService = fileContentServices.quarantine();
            IngestionQuarantineService.QuarantineResult quarantineCopy = quarantineService.quarantine(file);
            INDEXING_LOG.warn("[INDEXING] Content guard rejected file and copied it to quarantine");
            return LocalDocsFileOutcome.failedFile(new IngestionLocalFailure(
                    file.toString(),
                    "content-guard",
                    "quarantine copy " + quarantineCopy.quarantined() + ": " + rejectionReason));
        } catch (IOException quarantineException) {
            log.warn("Failed to quarantine invalid content (exception type: {})",
                    quarantineException.getClass().getSimpleName());
            return LocalDocsFileOutcome.failedFile(
                    failureFactory.failure(file, "quarantine-write", quarantineException));
        }
    }

// processExcludedPage — expected contract for “previously indexed URL is no longer indexable”:
    private LocalDocsFileOutcome processExcludedPage(MarkerContext markerContext, boolean requiresFullReindex) {
        try {
            if (requiresFullReindex) {
                storage.hybridVector().deleteByUrl(markerContext.collectionKind(), markerContext.url()); // expected cleanup (absent in quarantine)
                ingestedFilePruneService.pruneObsoleteLocalStateAfterReplacement(
                        markerContext.url(), markerContext.priorIngestionRecord().orElse(null), List.of());
            }
        } ...
        markFileIngested(markerContext.url(), new FileIngestionRecord(..., markerContext.collectionName(), List.of())); // also absent in quarantine
    }

Explanation

• For a previously-ingested file whose contents changed, inspectExistingMarker sets requiresFullReindex = true and markerContext is available in prepare() before the content guard runs.
• If the guard rejects the updated file, prepare() returns early via deferred(() -> quarantineRejectedFile(file, reason)) and throws away requiresFullReindex and markerContext.
• quarantineRejectedFile only copies the file to quarantine and returns failedFile; it never performs the cleanup that other “URL no longer indexable” paths perform.
• Result: Qdrant points (and local parsed-chunk state) for the URL remain from the prior good ingestion, while the on-disk mirrored file (and upstream URL) now serve guard-rejected content. Retrieval uses stored doc_content as both the citation snippet and RAG prompt text, but the citation url points to the upstream page now serving the rejected body.

Codebase Inconsistency

processExcludedPage handles the same high-level situation (“previously indexed URL is no longer indexable”) by deleting vectors, pruning local state, and marking the file ingested with empty chunkHashes. The guard-rejection/quarantine path skips all three steps.

Failing Test

// (New tests added) LocalDocsFileIngestionProcessorTest.java
// - shouldNotLeaveStaleVectorsWhenPreviouslyIngestedFileIsGuardRejected
// - shouldNotLeaveStaleVectorsWhenQuarantineWriteFailsForPreviouslyIngestedFile

Failing output on current branch (shows cleanup not invoked):

Wanted but not invoked:
hybridVectorService.deleteByUrl(
    <any com.williamcallahan.javachat.service.QdrantCollectionKind>,
    <any string>
);
...
However, there was exactly 1 interaction with this mock:
hybridVectorService.resolveCollectionName(
    DOCS
);

Recommended Fix

Route the guard-rejection branch through the same cleanup sequence processExcludedPage uses, while preserving the current failedFile outcome:

• Capture markerContext and requiresFullReindex before returning the deferred quarantine task.
• In quarantineRejectedFile (or a new helper invoked alongside it), when requiresFullReindex is true run:
  • deleteByUrl
  • pruneObsoleteLocalStateAfterReplacement(..., List.of())
  • markFileIngested(..., chunkHashes = List.of())
• Make the forensic quarantine copy best-effort: log failures but still run cleanup so index integrity does not depend on quarantine write succeeding.

History

This bug was introduced in commit 1263e64. The commit ("fix(search): preserve exact Javadoc citations", Jul 18 11:30) removed the earlier unconditional prune-before-guard behavior and deferred cleanup to post-storage paths that the guard-rejection branch exits before reaching, leaving previously-indexed vectors/state intact when the file becomes guard-rejected.