feat(navigator): hero split layout, tree persistence, multi-language syntax, shallow retention, and docs overhaul

ARCHITECTURE.md

@@ -74,9 +74,11 @@ flowchart TD
             TopoHelpers["helpers.py"]
         end
 
-        NavigatorSrv["Codebase Navigator Service (navigator.py)\n3-Pane Tree, AST Outline & Impact Engine"]
+        NavigatorSrv["Codebase Navigator Service (navigator.py)\nHero Tree, AST Outline & Impact Engine"]
+        OmniSearchSrv["Omni-Search Command Palette (omni_search.py)\nFuzzy Repo, File & Symbol Ranking"]
+        FileReaderSrv["Source & Doc Reader (file_reader.py)\nFull Code & Markdown Content Delivery"]
         LocalStorageSrv["Local Storage Service (local_storage.py)\nPath Traversal Defense & File Trees"]
-        GitMgr["Universal Shallow Git Ingestion (git_manager.py)"]
+        GitMgr["Universal Shallow Git Ingestion (git_manager.py)\nEphemeral & Persistent Clones"]
         Embeddings["FastEmbed Engine (embeddings.py)\nDense (384d) + Sparse BM25"]
         Search["Hybrid & RRF Search (search.py)"]
         Poller["Auto-Sync Poller Daemon (poller.py)"]
@@ -94,6 +96,7 @@ flowchart TD
         Chroma["ChromaDB (Embedded/Remote)"]
         end
         LocalStorageDir[("Managed Local Storage\n(DATA_DIR/storage)")]
+        PersistentRepoDir[("Persistent Shallow Clones\n(DATA_DIR/repos/{repo_name})")]
     end
 
     Claude -->|Authorization: Bearer cc_... or JWT| SSE
@@ -352,26 +355,19 @@ flowchart TD
 
 ---
 
-### 13. High-Performance 3-Pane Codebase Navigator (`app/services/navigator.py` & `app/api/routers/navigator.py`)
-- **Architectural Motivation**: Replaces legacy 2D graph canvases with a structured, ultra-fast 3-pane navigation paradigm designed for instant codebase comprehension, symbol discovery, and architectural impact analysis.
-- **Three-Pane Layout Topology**:
-  - **Pane 1: Files & Modules (`NavigatorTree.tsx`)**:
-    - Queries `GET /admin/api/navigator/tree?repo=...`.
-    - Recursively organizes `indexed_files` into hierarchical directory trees with aggregate metrics (total symbol counts, detected API routes per folder/file).
-    - Features instant search filtering (auto-expanding ancestor directories), Expand All / Collapse All controls, and active file highlighting.
-  - **Pane 2: Symbols & Routes (`NavigatorOutline.tsx`)**:
-    - Queries `GET /admin/api/navigator/file-outline?filepath=...&repo=...`.
-    - Retrieves syntax-aware AST symbols from `ast_symbols` along with associated REST routes from `api_routes`.
-    - Provides category chip filtering (`All`, `Functions`, `Classes`, `Routes`) and real-time symbol search.
-    - Displays symbol kinds (function, class, struct, interface), start/end line numbers, and route method badges (`POST`, `GET`, etc.).
-  - **Pane 3: Code Intelligence & Impact Inspector (`NavigatorInspector.tsx`)**:
-    - Queries `GET /admin/api/navigator/symbol-impact?symbol_id=...` or `?name=...&filepath=...`.
-    - Aggregates multi-source intelligence from `ast_symbols`, `ast_relationships`, and `api_routes`.
-    - **4-Metric Impact Summary**: Count of incoming callers, outgoing callees, imported modules, and language scope.
-    - **API Route Mapping Card**: Method badge (`POST`, `GET`, `PUT`, `DELETE`), path pattern (`/v1/chat/completions`), and framework tag (`FastAPI`, `Express`, `Gin`, etc.).
-    - **Signature & Docstrings**: Syntax-highlighted code block and extracted documentation.
-    - **Interactive Relationship Cards**: Clickable caller cards with filename, symbol name, and line numbers. Clicking a caller triggers bidirectional navigation (updates tree selection, switches file outline, and activates the caller symbol).
-    - **Copy Permalink**: Generates and copies clean permalinks with file paths and line ranges.
+### 13. High-Performance Codebase Navigator & Omni-Search (`app/services/navigator.py`, `app/services/omni_search.py`, `app/services/file_reader.py`)
+- **Architectural Motivation**: Provides an ultra-fast, IDE-grade codebase comprehension and exploration system with split hero layout, instant command palette search, syntax-highlighted code viewing, and deep AST symbol intelligence.
+- **Hero Split-View Layout Topology**:
+  - **Left Sidebar: Dual-Tab Directory Tree & Symbol Outline**:
+    - **Files Tab (`NavigatorTree.tsx`)**: Queries `GET /admin/api/navigator/tree?repo=...`. Recursively structures `indexed_files` into hierarchical trees with aggregate symbol counts and API routes. Features instant search filtering, Expand All / Collapse All controls, active file highlighting, and per-repository directory expansion persistence via `sessionStorage`.
+    - **Symbols Tab (`NavigatorOutline.tsx`)**: Queries `GET /admin/api/navigator/file-outline?filepath=...&repo=...`. Displays AST symbol declarations from `ast_symbols` and REST routes from `api_routes` with category filtering (`All`, `Functions`, `Classes`, `Routes`), signature previews, and search.
+  - **Center Hero Viewport: Code Viewer, Doc Reader & Intelligence**:
+    - **Syntax-Aware Source Code Viewer (`NavigatorCodeViewer.tsx`)**: Queries `GET /admin/api/navigator/file-content?filepath=...&repo=...` through `file_reader.py`. Features Prism-powered syntax highlighting across 8+ languages (C#, Python, JavaScript/TypeScript, C++, Go, Rust, SQL, COBOL, JSON, YAML, etc.), line numbering, target line range highlights (`targetStartLine`-`targetEndLine`), permalink copying, and collapsible caller impact drawer.
+    - **Documentation & Markdown Reader (`NavigatorDocReader.tsx`)**: Renders markdown files with GitHub Flavored Markdown (GFM), automatic table normalization (bridging blank lines and auto-inserting missing separator rows), and interactive Mermaid diagram rendering (`flowchart`, `sequenceDiagram`, `classDiagram`, `erDiagram`, etc.).
+    - **Code Intelligence & Impact Inspector (`NavigatorInspector.tsx`)**: Queries `GET /admin/api/navigator/symbol-impact?symbol_id=...`. Immediately accessible and clickable upon file selection without requiring manual outline navigation. Aggregates caller/callee counts, API route mappings, method signatures, docstrings, and cross-file jump navigation.
+  - **Omni-Search Command Palette (`NavigatorOmniSearch.tsx` / `Ctrl+K`)**:
+    - Queries `GET /admin/api/navigator/omni-search?query=...&repo=...`.
+    - Real-time fuzzy ranked matching across repository aliases, file paths, and AST symbols with hotkey triggers, badge indicators, and instant keyboard selection.
 - **Multi-Density Layout Engine & Persistent UX**:
   - `Balanced`: Default balanced layout optimized for standard desktop viewports.
   - `Compact`: High-density IDE layout reducing font sizes and padding for large file trees and complex outlines.
@@ -400,6 +396,7 @@ erDiagram
         datetime last_synced
         int enabled
         int auto_sync
+        int keep_shallow "0 = Ephemeral (delete on sync), 1 = Retain shallow clone on disk"
         string webhook_secret
         datetime added_at
     }

DEVELOPER_DOCS.md

@@ -90,24 +90,24 @@ cd contextcortex
 
 ### Backend Tests (Python Pytest & Coverage)
 ```bash
-# Run all backend unit and integration tests (277 tests)
+# Run all backend unit and integration tests (522+ tests)
 pytest -v
 
-# Run backend tests with code coverage report (88% coverage baseline)
+# Run backend tests with code coverage report (88%+ coverage baseline)
 pytest -v --cov=app --cov-report=term-missing
 ```
 
 ### Frontend Tests (React / TypeScript)
 ```bash
 cd frontend
 
-# Run unit and component tests via Vitest (82 tests)
+# Run unit and component tests via Vitest (315 tests across 32 test files)
 npm test
 
-# Run component tests with code coverage (87% line coverage)
+# Run component tests with code coverage
 npm run test:coverage
 
-# Run end-to-end user journey tests via Playwright (26 user journeys)
+# Run end-to-end user journey tests via Playwright
 npx playwright test
 ```
 
@@ -138,6 +138,7 @@ npm run docs:preview
 | :--- | :--- | :--- |
 | `DATABASE_URL` | SQLAlchemy connection string (e.g. `postgresql+psycopg://...` or `sqlite:///...`) | `sqlite:////app/data/index_cache.db` |
 | `LOCAL_STORAGE_PATH` | Storage directory for managed local storage file uploads | `/app/data/storage` |
+| `PERSISTENT_REPOS_DIR` | Storage directory for retained shallow git clones (`keep_shallow`) | `/app/data/repos` |
 | `AUTH_ENABLED` | Enable MCP 2026-07-28 OAuth 2.1 & API Key RBAC | `false` |
 | `VECTOR_STORE_PROVIDER` | Vector database backend (`pgvector`, `qdrant`, or `chroma`) | `qdrant` |
 | `VECTOR_STORE_MODE` | Vector store mode (`embedded` or `remote`) | `embedded` |
@@ -179,16 +180,18 @@ contextcortex/
 │       ├── indexing/      # Git/local syncers, file processor, state notifications
 │       ├── topology/      # Graph topology builder, node details, BFS helpers
 │       ├── vector_store/  # pgvector, Qdrant, and ChromaDB pluggable vector store implementations
-│       ├── navigator.py   # High-performance 3-pane codebase tree, outline & impact intelligence
+│       ├── navigator.py   # High-performance codebase tree, outline & impact intelligence
+│       ├── omni_search.py # Unified fuzzy command palette search for files, repos, and symbols
+│       ├── file_reader.py # Safe full-file source and markdown content resolution from disk
 │       ├── local_storage.py# Safe path resolution, disk file persistence, and tree inspection
-│       ├── git_manager.py # Ephemeral shallow git clone, token masking, permalinks
+│       ├── git_manager.py # Shallow git clone (ephemeral & persistent), token masking, permalinks
 │       ├── embeddings.py  # FastEmbed dense (384d) & sparse BM25 multi-vector engine
 │       ├── search.py      # Hybrid search & Reciprocal Rank Fusion (RRF) reranker
 │       ├── poller.py      # Background scheduled repo SHA poller daemon
 │       ├── adr.py         # MADR / Nygard format ADR ingestion and lifecycle
 │       ├── architecture.py# Codebase entry point, language distribution synthesis
 │       └── logger.py      # In-memory 500-event ring buffer diagnostic logger
-├── tests/                 # Backend pytest test suite (430+ tests, 88% coverage)
+├── tests/                 # Backend pytest test suite (522+ tests, 88%+ coverage)
 │   ├── backend/           # Unit and integration test modules
 │   ├── test_navigator_router.py      # REST navigator endpoints tests
 │   ├── test_navigator_service.py     # Tree, outline, and impact service tests

Dockerfile

@@ -7,6 +7,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
     git \
     libpq-dev \
     curl \
+    ripgrep \
     && rm -rf /var/lib/apt/lists/*
 
 COPY requirements.txt .

README.md

@@ -14,10 +14,10 @@ A high-performance, multi-repo Model Context Protocol (MCP) server providing **s
 | ![Overview](docs/assets/desktop_overview.png) | ![Git Repos](docs/assets/desktop_git-repos.png) |
 | **Codebase Navigator** | **Search & Inspector** |
 | ![Codebase Navigator](docs/assets/desktop_codebase-navigator.png) | ![Search](docs/assets/desktop_search-inspector.png) |
-| **Local Paths** | **Settings** |
-| ![Local Paths](docs/assets/desktop_local-paths.png) | ![Settings](docs/assets/desktop_settings.png) |
-| **Diagnostics & Logs** | |
-| ![Diagnostics](docs/assets/desktop_diagnostics.png) | |
+| **Files & Storage** | **Settings** |
+| ![Files & Storage](docs/assets/desktop_local-paths.png) | ![Settings](docs/assets/desktop_settings.png) |
+| **Diagnostics & Logs** | **Add Repository Modal** |
+| ![Diagnostics](docs/assets/desktop_diagnostics.png) | ![Add Repository Modal](docs/assets/desktop_add-repo-modal.png) |
 
 </details>
 
@@ -59,12 +59,13 @@ A high-performance, multi-repo Model Context Protocol (MCP) server providing **s
   - **Qdrant**: High-scale vector engine supporting dense + sparse BM25 multi-vectors with Reciprocal Rank Fusion (RRF) in both embedded disk and remote server modes.
   - **ChromaDB**: Lightweight, zero-dependency embedded disk or remote vector store with automatic fallback.
   - **Dynamic Backend Switching**: Test and switch vector backends live from the Settings UI or REST API without server restarts.
-- **Universal Git Provider Support**:
+- **Universal Git Provider Support & Persistent Shallow Retention**:
   - Ingest repositories from **any source where Git lives**: **GitHub**, **GitLab (Cloud, Enterprise & Self-Hosted)**, **Gitea & Forgejo**, **Bitbucket (Cloud & Server)**, and **Generic Git HTTP/HTTPS**.
+  - **Configurable Shallow Copy Retention (`keep_shallow`)**: Choose between **ephemeral shallow clones** (immediate deletion after indexing to save storage) or **persistent shallow clones** (`--depth 1` retained on disk under `/app/data/repos/{repo_name}` on the cache volume). Persistent copies power instant full source code inspection in Code Navigator and accelerated incremental delta syncing (`git fetch --depth 1` + `git reset --hard`).
+  - **Full-Screen React Portals for Modals**: Add Repository modal, Webhook setup modal, and Sync Drawer render cleanly via React Portals (`createPortal(..., document.body)`), eliminating containment clipping from glassmorphism cards.
   - **Provider-Aware Permalinks**: Automatically generates exact deep-links for code results (`/blob/`, `/-/blob/`, `/src/branch/`, `/src/commit/`, `/src/#lines-`).
   - **Custom Git Host Credential Vault**: Register per-host tokens and authentication types for private internal domains (e.g. `gitlab.company.internal` or `http://git.lan:3000`).
 - **AST-Aware Code Chunking (Tree-sitter)**: Understands syntax structures across Python, TypeScript/JavaScript, Go, Rust, C#, C++, Java, Ruby, PHP, and more. Chunks along class, method, and function boundaries with exact line numbers and symbol names.
-- **Ephemeral Repository Ingestion**: Authenticated shallow clones (`--depth 1`) extract AST symbols and hybrid vectors and **immediately remove the cloned repository from disk** to conserve storage.
 - **Multi-Tier Git Authentication Hierarchy**:
   1. Per-repository override token & optional username.
   2. Domain-level Custom Git Host Vault (`git_host_credentials`).
@@ -80,20 +81,21 @@ A high-performance, multi-repo Model Context Protocol (MCP) server providing **s
   - Granular multi-dimensional filtering by `source_type` (`all`, `git`, `monitored_path`, `local_storage`), `repo_name`, `path_prefix`, and `file_extension`.
   - Flexible granularity (`summary` for totals and status; `detailed` for hierarchical file trees).
 - **Fast Deterministic Symbol Lookup**: Built-in symbol table (`ast_symbols`) powers instantaneous symbol searches (`find_symbol`) and file outlines (`get_file_outline`) without token bloat.
-- **High-Performance 3-Pane Codebase Navigator**:
-  - **Pane 1 (Files & Modules)**: Virtualized folder & file tree hierarchy with symbol/route counts, instant filtering, and one-click expand/collapse.
-  - **Pane 2 (Symbols & Routes)**: Language-aware AST symbol declarations with category chip filtering (`All`, `Functions`, `Classes`, `Routes`), signature previews, and search.
-  - **Pane 3 (Code Intelligence & Impact)**: Deep architectural intelligence displaying incoming callers, outgoing callees, imported modules, REST API route mappings (`POST`, `GET`, etc.), signature code blocks, docstrings, and one-click caller navigation jump.
-  - **Integrated Document & Markdown Reader**: Formatted Markdown and raw source table viewer for reading full documentation (`.md`, `.txt`) directly within Pane 3.
+- **High-Performance Codebase Navigator & Omni-Search**:
+  - **Omni-Search Command Palette (`Ctrl+K` / `Cmd+K`)**: Unified command palette searching across repositories, file paths, and AST symbols with live fuzzy scoring and keyboard navigation.
+  - **Dual-Pane Sidebar (Files & Symbols)**: Tabbed sidebar separating file directory hierarchy and AST symbol outline. Preserves folder expansion state per-repository across browser sessions (`sessionStorage`).
+  - **Hero Viewport (Full Source Code & Documentation)**:
+    - **Syntax-Highlighted Source Code Viewer**: High-fidelity syntax highlighting across 8+ languages (C#, Python, JavaScript/TypeScript, C++, Go, Rust, SQL, COBOL, JSON, YAML, etc.) with line numbers, highlighted target symbol line ranges, permalink copy, and collapsible caller impact drawer.
+    - **Markdown Document Reader**: Automatic table auto-normalization (bridging gaps and synthesizing separator rows for loose Markdown) and interactive Mermaid diagram rendering.
+    - **Instant Intelligence & Impact Tab**: Caller/callee analysis, route mappings, AST signatures, and cross-file jump navigation, immediately clickable and populated upon file selection without requiring manual outline navigation.
   - **Customizable Layout Density & Mobile Responsiveness**: Persisted `Compact` (IDE density), `Balanced` (default), and `Spacious` (cards) modes with responsive vertical stacking and zero horizontal overflow across devices.
 - **Diagnostic Logging & Observability**: In-memory ring buffer (500 events) capturing server warnings, errors, indexing lifecycle events, and expandable stack traces with a REST API (`/admin/api/logs`).
 - **Multi-Theme Engine & Modern Tabbed Web Dashboard (`/admin/`)**:
   - **Appearance & Theme Settings**: Instant zero-latency switching between 4 distinct dark and light themes (**Deep Ocean**, **Midnight Blue**, **Lavender Haze**, and **Amber Warmth**) with live palette swatches and browser persistence.
   - **Overview**: Real-time stats, vector counts, AST symbols, model specs, topic tag cloud, and manual full reindexing trigger.
-  - **Codebase Navigator**: High-performance 3-pane architectural file tree, AST symbol outline, and code impact/route inspector.
-  - **Git Repositories**: Register repos across GitHub, GitLab, Gitea, Bitbucket, or Generic Git, trigger shallow clone syncs, inspect commit SHAs, and manage sources.
-  - **Local Paths**: Monitor local workspaces and notes vaults with recursive directory scanning and filesystem browser modal.
-  - **Local Storage**: Managed file explorer, direct file upload modal with folder categorization, and file preview/replacement.
+  - **Codebase Navigator**: Hero split layout with Omni-Search palette, folder tree persistence, syntax-highlighted code viewer, Mermaid documentation reader, and instant intelligence inspection.
+  - **Git Repositories**: Register repos across GitHub, GitLab, Gitea, Bitbucket, or Generic Git, configure shallow copy retention, trigger syncs, inspect commit SHAs, and manage sources.
+  - **Files & Storage**: Unified explorer managing monitored local directory vaults and managed local file storage with upload modal and file preview.
   - **Ingestion Catalog**: Unified multi-source explorer with source type filters, repository lookup, and file listings.
   - **Search & Inspector**: Interactive live hybrid search tester with RRF score previews, target type toggle (Code vs Docs), and syntax highlighted results.
   - **Settings**: Vector Database manager (pgvector, Qdrant, & ChromaDB switcher & connection tester), LiteLLM Model Discovery with dynamic categorized model dropdowns (Embeddings, Vision OCR, and Chat models), multi-provider token cards, GitHub rate limit monitor, and interactive Custom Git Host Credential Vault table/modal.