Skip to content

Add readable multi-model filesystem vector views - #26

Open
isomorphisms wants to merge 5 commits into
clarify-inclusion-and-browser-contextfrom
readable-vector-views
Open

Add readable multi-model filesystem vector views#26
isomorphisms wants to merge 5 commits into
clarify-inclusion-and-browser-contextfrom
readable-vector-views

Conversation

@isomorphisms

Copy link
Copy Markdown
Owner

What this makes real

  • Adds the requested views/retrieved-from-the-web/{text,images} and views/organizing-the-information/{categories,vector-spaces} layout.
  • Implements an exact C99 vector backend whose authoritative vectors are fixed-width plain text: 15 ASCII bytes per signed scientific Float32 plus one delimiter byte. The byte offset is (row * dimensions + coordinate) * 16.
  • Keeps the little-endian Float32 file only as a rebuildable speed cache. query-text works without it, compile-cache recreates it, column performs direct coordinate seeks, and check proves text/cache equality.
  • Pins two deliberately different Hugging Face views by immutable commit, size, SHA-256, preprocessing, and output dimension:
    • minishlab/potion-base-2M ONNX: 64 coordinates, static lookup/mean/normalize.
    • mixedbread-ai/mxbai-embed-xsmall-v1 INT8 ONNX: 384 coordinates, transformer/mean/normalize.
  • Adds verified fetch/materialization commands and copies each exact model manifest beside its vector-space collections.
  • Runs both actual ONNX models through embed → filesystem build → cached query → text-only query in CI.

Verification

  • Warning-clean C99 build and smoke suite.
  • AddressSanitizer and UndefinedBehaviorSanitizer pass (leak detection disabled because the work sandbox runs under ptrace).
  • Corrupt text, corrupt cache, missing cache, strict-manifest, empty-index, dimension, zero-norm, and score-overflow cases covered.
  • 10,000 × 384 scale run produced exactly 61,440,000 readable bytes and 15,360,000 cache bytes; cached and text-only rankings were byte-identical.
  • A distinct semantic query about closed nondegenerate two-forms retrieved the symplectic-geometry fixture first with both models.

Boundary

This completes the filesystem/index/model-provenance layer without a vector database. The ONNX adapter is deliberately under experiments/; moving selected inference into the phone-resident Android browsing loop remains a separate runtime step.

Stacked on #24.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant