Skip to content

Add filesystem Float32 vector backend - #17

Draft
isomorphisms wants to merge 4 commits into
idric-durable-file-iofrom
flat-file-vector-index
Draft

Add filesystem Float32 vector backend#17
isomorphisms wants to merge 4 commits into
idric-durable-file-iofrom
flat-file-vector-index

Conversation

@isomorphisms

Copy link
Copy Markdown
Owner

Summary

  • add a dependency-free C99 ib-vector-index backend using exact scans over row-major little-endian float32
  • keep the interface composable and text-first: TSV build rows on stdin, a text query on stdin, and TSV ranked results on stdout
  • store inspectable format.txt and generated ID text beside immutable .f32 generations under state/indexes/vectors/<collection>/<model>
  • switch generations by replacing format.txt last, so an interrupted rebuild cannot select partial vector bytes
  • add an Idriç backend record and command lowering so this implementation can later be replaced without changing canonical records or callers
  • classify the manifest and ID files as readable derived metadata while keeping vector bytes out of generic text inspection

Why exact first

IB's workbench target is 10,000 URLs. At 384 dimensions the vectors occupy 15,360,000 bytes and a query performs 3.84 million multiply-adds. This remains small enough for a sequential exact reference backend; HNSW graph state and tuning are deferred until phone measurements show a need.

Precision

The stored values, normalization, accumulation, and scores are all 32-bit float. The existing compiler's lack of an Idriç Float primitive does not force rebuildable index state to double: the native backend owns the numeric buffer and Idriç owns only its typed specification.

Checks

  • warning-clean C99 host build
  • deterministic build/query/validation smoke test
  • ASan/UBSan smoke run locally
  • 10,000 × 384 CI fixture requiring exactly 15,360,000 vector bytes
  • pinned Android NDK cross-builds for arm64-v8a and armeabi-v7a, each stripped and capped below 100 KiB
  • Idriç smoke assertions for backend selection, scalar, derived path, readable metadata, and non-text vector bytes

This is stacked on idric-durable-file-io so it uses the real filesystem store without colliding with that work. It does not merge or modify PR #9.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant