Skip to content

[C API] Expose leanvec ood - #358

Open
ethanglaser wants to merge 35 commits into
mainfrom
dev/eglaser-leanvec-ood-capi
Open

[C API] Expose leanvec ood#358
ethanglaser wants to merge 35 commits into
mainfrom
dev/eglaser-leanvec-ood-capi

Conversation

@ethanglaser

@ethanglaser ethanglaser commented Jul 28, 2026

Copy link
Copy Markdown
Member

Summary

Exposes LeanVec out-of-distribution (OOD) training to the C API. Callers can train dimensionality-reduction matrices from a data sample plus a sample of queries, then build an index that uses those matrices instead of computing PCA matrices at build time.

API

// Train matrices. Pass num_queries=0 / x_q=NULL for in-distribution (PCA).
svs_leanvec_training_data_h svs_leanvec_training_data_build(
    size_t dim, size_t num_vectors, const float* x,
    size_t num_queries, const float* x_q,
    size_t leanvec_dims, svs_error_h out_err);

void svs_leanvec_training_data_free(svs_leanvec_training_data_h);

// LeanVec storage using pre-trained matrices; leanvec_dims comes from the
// training data. Storage holds its own reference, so the handle may be freed
// as soon as this returns.
svs_storage_h svs_storage_create_leanvec_trained(
    svs_leanvec_training_data_h training_data,
    svs_data_type_t primary, svs_data_type_t secondary,
    svs_error_h out_err);

Training data is attached at storage construction rather than via an index-builder setter (per review): leanvec_dims has a single source of truth, storage configs stay immutable and safely shareable across builders, and misuse on a non-LeanVec
storage is a compile-time error instead of a runtime one.

Tests

c_api_index.cpp: OOD build + search, in-distribution pre-trained build + search (freeing the training data before build, to exercise shared ownership of the matrices), and NULL-training-data rejection. All sections no-op gracefully on builds or hardware without LeanVec support.

Notes

  • Renamed internal StorageLeanVec::lenavec_dimsleanvec_dims.
  • LeanVec paths are compile- and ABI-verified locally but not runtime-verified: the C bindings' pinned SVS_URL predates MemoryBreakdown (the C++ bindings were bumped in Update SVS_URL to nightly with get_memory_breakdown #357, the C ones weren't), and the only nightly that has it is LTO-built and won't link against local GCC 11.4. Relying on CI with private sources here.

rfsaliev and others added 30 commits March 4, 2026 15:22
Add `svs_index_load()` and `svs_index_save()` API implementation for
static Vamana index
Done:

- [x] Create dynamic index with specified block size (default block size
should be supported)
- [x] Initialized with a dataset and labels list
- [x] Add labeled vectors to a dynamic index
- [x] Remove vectors by labels
- [x] Check if a label exists
- [x] Compute distance for label
- [x] Get vector by label
- [x] Consolidate/compact dynamic index
- [x] Implement Save/Load
Adds `svs_index_get_num_threads` / `svs_index_set_num_threads` to the C
API, enabling dynamic inspection and resizing of the search threadpool
after index construction.

### ThreadPoolBuilder
- Added `get_threads_num()` — delegates to the custom pool's `size()` op
when `kind == CUSTOM`, otherwise returns the stored count
- Added `resize(n)` — updates stored thread count; throws
`std::invalid_argument` for `n == 0`, `SINGLE_THREAD`, or `CUSTOM` kinds
(surfaced as `SVS_ERROR_INVALID_ARGUMENT` through `wrap_exceptions`)

### Index wrappers (`index.hpp`)
- `Index` stores a `ThreadPoolBuilder`; `get_num_threads()` is
pure-virtual — implemented in `IndexVamana` and `DynamicIndexVamana` by
delegating to the wrapped `svs::Vamana` / `svs::DynamicVamana` instance,
so the value reflects actual runtime state
- `set_num_threads(n)` calls `pool_builder.resize(n)` then rebuilds and
installs the threadpool via `set_threadpool()`

### C API (`svs_c.cpp` / `svs_c.h`)
- Both entry points validate `index->impl` non-null before dereferencing
(consistent with existing handle-check pattern)
- Public header documents supported kinds and expected error codes for
unsupported configurations
Resolve cmake version compatibility issue caused by using
DOWNLOAD_EXTRACT_TIMESTAMP which is introduced in v.3.24

This PR fixes #317
…306)

This pull request introduces a comprehensive C API test suite for the
SVS project, leveraging the Catch2 testing framework. It adds new test
files covering all major C API functionalities, integrates automated
test building and execution into the CMake build system, and improves
error handling and testability for dynamic index operations.

**C API Test Infrastructure and Test Coverage:**

* Added a new directory of C API tests using Catch2, with individual
test files for error handling, algorithm configuration, storage, search
parameters, index building, and dynamic index operations.

**Dynamic Index Error Handling:**

* Refactored `svs_index_dynamic_delete_points` to improve error handling.
- Introduced `svs_id_filter_interface`  to define filtering operations.
- Implemented `svs_index_search_topK` to support an optional ID filter
for search operations.
- Updated existing search functions to use the new filtered search
capabilities.
- Added a new source file `filtered_search.hpp` containing the logic for
filtered top-K search.
- Modified existing samples and tests to demonstrate and validate the
new filtering functionality.
- Marked the previous `svs_index_search` function as deprecated,
directing users to use `svs_index_search_topK` instead.
…n) (#354)

## Summary

Exposes memory accounting in the **C API** for the Valkey-search
integration:

- `svs_index_get_memory_usage(index, size_t* out_bytes, err)` — total
allocated bytes.
- `svs_index_get_memory_breakdown(index, svs_memory_breakdown_t* out,
err)` — `{graph_bytes, data_bytes, metadata_bytes}` component split.
~~- `svs_index_element_size(index, size_t* out_bytes, err)` — bytes per
stored vector.~~ (keep at data level)

All follow the existing C API conventions (out-param + `svs_error_h`,
`wrap_exceptions`), matching the Phase-A design in the memory-accounting
contract (intel-innersource #333).

## Layers

- **C API** (`bindings/c`): the three functions +
`svs_memory_breakdown_t` in `svs_c.h`; interface virtuals + concrete
overrides in `src/index.hpp`; impls in `src/svs_c.cpp`.
- **Core / orchestrator**: brings in `get_memory_breakdown()`
(`MemoryBreakdown` struct + capacity-based
`svs::data::detail::dataset_allocated_bytes` helper) on `VamanaIndex` /
`MutableVamanaIndex` and through the orchestrator, plus an
`element_size()` accessor parallel to `dimensions()`. This mirrors the
approved public PR #345 so the C API can build and test standalone; once
#345 lands on `dev/c-api`, this reduces to just the C API layer.

## Tests

`bindings/c/tests/c_api_index.cpp` (static) and
`c_api_dynamic_index.cpp` (dynamic): usage > 0, breakdown total ==
usage, `graph_bytes`/`data_bytes` > 0 (metadata > 0 for dynamic),
`element_size == sizeof(float) * dimensions`, and null-arg handling.
Both test cases pass (84 / 166 assertions).

Related: builds on #345; memory-accounting
contract in intel-innersource #333 / #326.
@ethanglaser ethanglaser changed the title Dev/eglaser leanvec ood capi Expose leanvec ood in C API Jul 28, 2026
@ethanglaser ethanglaser changed the title Expose leanvec ood in C API [C API] Expose leanvec ood Jul 28, 2026

@rfsaliev rfsaliev left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

API concerns

Comment thread bindings/c/include/svs/c_api/svs_c.h Outdated
/// time. Pass NULL to clear a previously attached training data.
/// @param out_err An optional error handle to capture errors
/// @return true on success, false on failure
SVS_API bool svs_index_builder_set_leanvec_training_data(

@rfsaliev rfsaliev Aug 3, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like the "training data" belongs to LeanVec storage kind only.
The proper API call should be connected to LeanVec storage only:
Preffered:

SVS_API svs_storage_h svs_storage_create_leanvec_pre_trained(
    svs_leanvec_training_data_h training_data, // leanvec_dims to be extracted from there
    svs_data_type_t primary,
    svs_data_type_t secondary,
    svs_error_h out_err /*=NULL*/
);

alternative:

SVS_API bool svs_storage_set_leanvec_training_data(
    svs_storage_h leanvec_storage,
    svs_leanvec_training_data_h training_data, // leanvec_dims to be extracted from there and compared with previously defined
    svs_error_h out_err /*=NULL*/
);

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated, but named it svs_storage_create_leanvec_trained

@ethanglaser
ethanglaser marked this pull request as ready for review August 4, 2026 23:28
@mergify

mergify Bot commented Aug 4, 2026

Copy link
Copy Markdown

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request

@ethanglaser
ethanglaser requested a review from rfsaliev August 5, 2026 18:46
rfsaliev and others added 2 commits August 10, 2026 08:29
#360 reopened directly
to C API branch

---------

Co-authored-by: Rafik Saliev <rafik.f.saliev@intel.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
@rfsaliev

Copy link
Copy Markdown
Member

@ethanglaser, it seems this PR contains number of changes out-of bindings/c directory.
To avoid potential conflict with the main branch, I would postpone this PR till dev/c-api be merged to the main branch.

@rfsaliev
rfsaliev force-pushed the dev/c-api branch 2 times, most recently from e0cbd2f to c57d2ad Compare August 24, 2026 10:27
Base automatically changed from dev/c-api to main August 24, 2026 12:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants