Evaluate a Float16 vector model and GPU index

[isomorphisms]

Question

Can IB use a Float16 embedding model and Float16 vector index to reduce model size, RAM, cache size, and query cost on the target 32-bit ARMv7 Android Go phone—especially through its PowerVR GPU—without materially damaging retrieval or category proposals?

Preserve the readable Float32 vectors as the reference representation. Float16 model weights/inference, Float16 stored embeddings, and Float32 versus Float16 accumulation are separate choices and must be measured separately.

Candidate execution paths

Compare at least:

Path	Stored vectors	Accumulator
CPU reference	Float32	Float32
GPU safe	Float16	Float32
GPU aggressive	Float16	Float16 partial sums, Float32 final reduction

The GPU path should keep the roughly 10,000 × 384 Float16 index resident while the text frontend is active when memory permits. Each query uploads only its 384 coordinates; scoring may return the Float32 score array or perform a second-stage top-k reduction on the GPU.

Do not assume the advertised PowerVR model or feature set. Record glGetString(GL_RENDERER) and glGetString(GL_VERSION), then test whether the required OpenGL ES 3.1 compute shader and Float16 storage/arithmetic path actually compile and run. A CPU fallback remains required.

Constraints

• Keep canonical text, source locators, accepted labels, corrections, and proposal provenance independent of the model.
• Treat every embedding generation and vector index as versioned, rebuildable derived state.
• Retain the pinned Float32 exact-search path as oracle and fallback.
• Record model id, immutable revision, dimensions, tokenizer/preprocessing, precision layout, and generator version.
• Do not silently label a Float16 score as identical to the Float32 score.
• Browsing and lexical retrieval must continue to work when the model or GPU path is absent, slow, or crashes.

Benchmark

Run the same corpus and queries through every path. Measure:

• model bytes, resident CPU/GPU memory, and cache bytes;
• cold single-query, warm single-query, and batch latency;
• GPU wake, upload, dispatch, synchronization, and readback costs;
• top-k agreement, recall/ranking changes, and category decision changes against Float32;
• determinism across clean rebuilds;
• sustained behavior on the actual phone.

The backend should be selected from measured device behavior, not paper specifications. A single cold query may favor the CPU even when warm or batched work favors the GPU.

Acceptance criteria

• A pinned Float16 candidate is identified with license and complete provenance.
• Float32 and Float16 artifacts cannot be confused in storage or proposal records.
• The phone reports the actual GL renderer/version and demonstrates or rejects the required compute path.
• Benchmarks cover cold, warm, and batched queries with synchronization included.
• Top-k agreement and classifier changes are compared directly with Float32.
• Any quality threshold is stated before choosing a default.
• Float16 remains a disposable implementation choice, not canonical browsing state.
• The result ends with an evidence-backed CPU/GPU backend-selection rule.