Skip to content

[C API] Implement memory estimation functions for index builders and tests - #364

Draft
rfsaliev wants to merge 3 commits into
mainfrom
rfsaliev/c-api-estimate-index-size
Draft

[C API] Implement memory estimation functions for index builders and tests#364
rfsaliev wants to merge 3 commits into
mainfrom
rfsaliev/c-api-estimate-index-size

Conversation

@rfsaliev

Copy link
Copy Markdown
Member

This pull request adds support for estimating the memory usage of an index (both static and dynamic Vamana) based on the builder configuration and number of vectors, and implements the necessary logic and API functions in the C bindings. It introduces new estimation methods in the C API, adds implementations for memory estimation in the builder and dispatcher classes, and updates data builders to support size estimation for various data types.

API Additions:

  • Added two new C API functions: svs_index_builder_estimate_memory and svs_index_builder_estimate_memory_dynamic to estimate the memory usage of an index and a dynamic index, respectively, based on the builder configuration and number of vectors.

Core Logic and Implementation:

  • Implemented estimate_memory_breakdown and estimate_memory_breakdown_dynamic methods in the IndexBuilder class to provide memory usage estimates for static and dynamic Vamana indexes.
  • Added dispatcher functions dispatch_vamana_memory_estimate and dispatch_dynamic_vamana_memory_estimate to calculate the memory breakdown for static and dynamic Vamana indexes, including graph, data, and metadata sizes. [1] [2]
  • Updated CMake configuration to include the new data_builder.cpp source file.

Data Builder Enhancements:

  • Added estimate_size methods to all data builder classes (SimpleDataBuilder, LVQDataBuilder, LeanVecDataBuilder, SQDataBuilder) to support size estimation for different data types. [1] [2] [3] [4] [5]
  • Created a new data_builder.cpp implementation file with logic for dispatching and estimating data sizes for various data builder specializations, including handling for blocked and non-blocked allocators.

Header and Dependency Updates:

  • Updated headers and includes to wire up the new estimation logic and expose the new API and dispatcher functions. [1] [2] [3] [4] [5] [6]

These changes collectively provide a robust mechanism for clients to estimate index memory requirements before actually building an index, improving usability and resource planning.

@rfsaliev

Copy link
Copy Markdown
Member Author

This PR contains number of workarounds/TODOs which should be resolved out-of the bindings/c directory.
To avoid potential conflicts, keeping the PR as 'Draft' until getting bindings/c on the main branch.

@rfsaliev
rfsaliev force-pushed the dev/c-api branch 2 times, most recently from e0cbd2f to c57d2ad Compare August 24, 2026 10:27
Base automatically changed from dev/c-api to main August 24, 2026 12:01
@rfsaliev
rfsaliev force-pushed the rfsaliev/c-api-estimate-index-size branch from cc12653 to 1d381ec Compare August 24, 2026 12:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant