[C API] Expose leanvec ood - #358
Open
ethanglaser wants to merge 35 commits into
Open
Conversation
Add `svs_index_load()` and `svs_index_save()` API implementation for static Vamana index
Done: - [x] Create dynamic index with specified block size (default block size should be supported) - [x] Initialized with a dataset and labels list - [x] Add labeled vectors to a dynamic index - [x] Remove vectors by labels - [x] Check if a label exists - [x] Compute distance for label - [x] Get vector by label - [x] Consolidate/compact dynamic index - [x] Implement Save/Load
Adds `svs_index_get_num_threads` / `svs_index_set_num_threads` to the C API, enabling dynamic inspection and resizing of the search threadpool after index construction. ### ThreadPoolBuilder - Added `get_threads_num()` — delegates to the custom pool's `size()` op when `kind == CUSTOM`, otherwise returns the stored count - Added `resize(n)` — updates stored thread count; throws `std::invalid_argument` for `n == 0`, `SINGLE_THREAD`, or `CUSTOM` kinds (surfaced as `SVS_ERROR_INVALID_ARGUMENT` through `wrap_exceptions`) ### Index wrappers (`index.hpp`) - `Index` stores a `ThreadPoolBuilder`; `get_num_threads()` is pure-virtual — implemented in `IndexVamana` and `DynamicIndexVamana` by delegating to the wrapped `svs::Vamana` / `svs::DynamicVamana` instance, so the value reflects actual runtime state - `set_num_threads(n)` calls `pool_builder.resize(n)` then rebuilds and installs the threadpool via `set_threadpool()` ### C API (`svs_c.cpp` / `svs_c.h`) - Both entry points validate `index->impl` non-null before dereferencing (consistent with existing handle-check pattern) - Public header documents supported kinds and expected error codes for unsupported configurations
Resolve cmake version compatibility issue caused by using DOWNLOAD_EXTRACT_TIMESTAMP which is introduced in v.3.24 This PR fixes #317
…306) This pull request introduces a comprehensive C API test suite for the SVS project, leveraging the Catch2 testing framework. It adds new test files covering all major C API functionalities, integrates automated test building and execution into the CMake build system, and improves error handling and testability for dynamic index operations. **C API Test Infrastructure and Test Coverage:** * Added a new directory of C API tests using Catch2, with individual test files for error handling, algorithm configuration, storage, search parameters, index building, and dynamic index operations. **Dynamic Index Error Handling:** * Refactored `svs_index_dynamic_delete_points` to improve error handling.
- Introduced `svs_id_filter_interface` to define filtering operations. - Implemented `svs_index_search_topK` to support an optional ID filter for search operations. - Updated existing search functions to use the new filtered search capabilities. - Added a new source file `filtered_search.hpp` containing the logic for filtered top-K search. - Modified existing samples and tests to demonstrate and validate the new filtering functionality. - Marked the previous `svs_index_search` function as deprecated, directing users to use `svs_index_search_topK` instead.
…n) (#354) ## Summary Exposes memory accounting in the **C API** for the Valkey-search integration: - `svs_index_get_memory_usage(index, size_t* out_bytes, err)` — total allocated bytes. - `svs_index_get_memory_breakdown(index, svs_memory_breakdown_t* out, err)` — `{graph_bytes, data_bytes, metadata_bytes}` component split. ~~- `svs_index_element_size(index, size_t* out_bytes, err)` — bytes per stored vector.~~ (keep at data level) All follow the existing C API conventions (out-param + `svs_error_h`, `wrap_exceptions`), matching the Phase-A design in the memory-accounting contract (intel-innersource #333). ## Layers - **C API** (`bindings/c`): the three functions + `svs_memory_breakdown_t` in `svs_c.h`; interface virtuals + concrete overrides in `src/index.hpp`; impls in `src/svs_c.cpp`. - **Core / orchestrator**: brings in `get_memory_breakdown()` (`MemoryBreakdown` struct + capacity-based `svs::data::detail::dataset_allocated_bytes` helper) on `VamanaIndex` / `MutableVamanaIndex` and through the orchestrator, plus an `element_size()` accessor parallel to `dimensions()`. This mirrors the approved public PR #345 so the C API can build and test standalone; once #345 lands on `dev/c-api`, this reduces to just the C API layer. ## Tests `bindings/c/tests/c_api_index.cpp` (static) and `c_api_dynamic_index.cpp` (dynamic): usage > 0, breakdown total == usage, `graph_bytes`/`data_bytes` > 0 (metadata > 0 for dynamic), `element_size == sizeof(float) * dimensions`, and null-arg handling. Both test cases pass (84 / 166 assertions). Related: builds on #345; memory-accounting contract in intel-innersource #333 / #326.
rfsaliev
requested changes
Aug 3, 2026
| /// time. Pass NULL to clear a previously attached training data. | ||
| /// @param out_err An optional error handle to capture errors | ||
| /// @return true on success, false on failure | ||
| SVS_API bool svs_index_builder_set_leanvec_training_data( |
Member
There was a problem hiding this comment.
It seems like the "training data" belongs to LeanVec storage kind only.
The proper API call should be connected to LeanVec storage only:
Preffered:
SVS_API svs_storage_h svs_storage_create_leanvec_pre_trained(
svs_leanvec_training_data_h training_data, // leanvec_dims to be extracted from there
svs_data_type_t primary,
svs_data_type_t secondary,
svs_error_h out_err /*=NULL*/
);alternative:
SVS_API bool svs_storage_set_leanvec_training_data(
svs_storage_h leanvec_storage,
svs_leanvec_training_data_h training_data, // leanvec_dims to be extracted from there and compared with previously defined
svs_error_h out_err /*=NULL*/
);
Member
Author
There was a problem hiding this comment.
Updated, but named it svs_storage_create_leanvec_trained
ethanglaser
marked this pull request as ready for review
August 4, 2026 23:28
ethanglaser
requested review from
ahuber21,
homksei,
ibhati,
mihaic and
yuejiaointel
as code owners
August 4, 2026 23:28
|
Tick the box to add this pull request to the merge queue (same as
|
#360 reopened directly to C API branch --------- Co-authored-by: Rafik Saliev <rafik.f.saliev@intel.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Member
|
@ethanglaser, it seems this PR contains number of changes out-of |
rfsaliev
force-pushed
the
dev/c-api
branch
2 times, most recently
from
August 24, 2026 10:27
e0cbd2f to
c57d2ad
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Exposes LeanVec out-of-distribution (OOD) training to the C API. Callers can train dimensionality-reduction matrices from a data sample plus a sample of queries, then build an index that uses those matrices instead of computing PCA matrices at build time.
API
Training data is attached at storage construction rather than via an index-builder setter (per review):
leanvec_dimshas a single source of truth, storage configs stay immutable and safely shareable across builders, and misuse on a non-LeanVecstorage is a compile-time error instead of a runtime one.
Tests
c_api_index.cpp: OOD build + search, in-distribution pre-trained build + search (freeing the training data before build, to exercise shared ownership of the matrices), and NULL-training-data rejection. All sections no-op gracefully on builds or hardware without LeanVec support.Notes
StorageLeanVec::lenavec_dims→leanvec_dims.SVS_URLpredatesMemoryBreakdown(the C++ bindings were bumped in Update SVS_URL to nightly with get_memory_breakdown #357, the C ones weren't), and the only nightly that has it is LTO-built and won't link against local GCC 11.4. Relying on CI with private sources here.