Skip to content

SDSTOR-20656: clear wbc chunk selector - #912

Open
nnastonen wants to merge 1 commit into
eBay:dev/v8.xfrom
nnastonen:SDSTOR-20656_clear_wbc_chunk_selector_v8
Open

nnastonen wants to merge 1 commit into
eBay:dev/v8.xfrom
nnastonen:SDSTOR-20656_clear_wbc_chunk_selector_v8

Conversation

@nnastonen

@nnastonen nnastonen commented Sep 10, 2026

Copy link
Copy Markdown
Contributor
  1. Create homestore with chunk selector
  2. Create btree1 and populate it
  3. Nodes get into wbc cache
  4. Destroy btree1 (wbc purge is not done, also volume's chunk bitmap is not cleared)
  5. Create btree2 and populate it
  6. Nodes get into wbc cache and we get cache corruption, duplicate inserts

This change makes chunk release asynchronous and defers returning the chunk to the free pool until cleanup is complete.

New flow:

  • detach the volume's chunk list from m_volume_chunks
  • run asynchronous cleanup for all the chunks
  • in cleanup, evict all allocated blkids for the chunks from WBC
  • [reset the chunks' block allocator state (bitmap)]
  • reinsert the chunks into m_per_dev_chunks

This PR also addresses this ticket SDSTOR-25404

@nnastonen
nnastonen force-pushed the SDSTOR-20656_clear_wbc_chunk_selector_v8 branch 3 times, most recently from 4ad7814 to e2d3391 Compare September 10, 2026 14:04
@nnastonen
nnastonen force-pushed the SDSTOR-20656_clear_wbc_chunk_selector_v8 branch from e2d3391 to 1d2aef4 Compare September 10, 2026 14:07
Comment thread src/CMakeLists.txt
${HOMESTORE_OBJECTS}
)
target_link_libraries(homestore ${COMMON_DEPS})
target_compile_definitions(homestore PRIVATE _PRERELEASE)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why we need this _PRERELEASE here? I think target_compile_definitions(test_index_chunk_selector PRIVATE _PRERELEASE) in tests/CmakeLists.txt is enough for the new test case

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because otherwise the code won't compile.

@JacksonYao287 JacksonYao287 Sep 14, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

adding _PRERELEASE unconditionally to the homestore library itself means every build (including production Release builds) will now permanently carry _PRERELEASE: extra fault-injection checkpoints, extra assert branches, and extra debug fields get baked into the production library. That's a much bigger behavioral change than "just make the new test compile." pls correct me if I misunderstand something.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right. I added _PRERELEASE just to make this compile and run. Please suggest a better place/way to enable _PRERELEASE as I didn't really find how it is enabled in 7.x. I presume it is not the same as Debug build.

virtual void recover(sisl::byte_view sb) = 0;

/// Remove entries for a specific chunk from wbc
virtual void evict_chunk_blkids(const Chunk& chunk) = 0;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIT: if this a time-consuming task, I would suggest make it return a sisl::async::task < void >, so that we can use co_wait to wait for the completion asyncronously?

@nnastonen nnastonen Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is a time-consuming task, we call it from HomeBlocks within iomanager.run_on_forget(), check the other half of this PR: eBay/HomeBlocks#179
We don't need to wait for it as it does all the job itself.

@JacksonYao287 JacksonYao287 Sep 14, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO, if we do nott need wait for it and it is a time-consuming task, then it would be better to return a sisl::async::task < void >, so that we can use detach to run this time-consuming task asyncronously and not block the calling thread

@nnastonen nnastonen Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

iomanager.run_on_forget() is also async, it does not block the caller and I am not a big fan of detach() personally. So this is basically a stylistic matter, unless there is something I don't know about sisl::async.
Please elaborate how this would be better for this particular case.

Comment thread src/lib/device/chunk.cpp
const auto cid = chunk_id();

for (blk_num_t blk_num{}; blk_num < total_blks; ++blk_num) {
blk_id blkid{blk_num, 1, cid};

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not quite sure if we need a lock to guarantee the thread safety of this function

@nnastonen nnastonen Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please elaborate? What exactly needs to be protected?
The chunk is not visible to chunk selector yet and SimpleCache seems to be thread safe.

@JacksonYao287 JacksonYao287 Sep 14, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

honestly this is just my intuition . if a thread is calling free_blk or reset blk_allocator, another thread is calling foreach_allocated_blk, not sure this will bring some thread safety issue.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing should touch (or even see) this chunk while chunk selector is in charge.

@xiaoxichen

Copy link
Copy Markdown
Collaborator

@szmyd @shosseinimotlagh Please if you can ensure the upgrade path and compatibility are properly discussed.

NuObject is moving to V7 mostly to intake the folly and co-routine removal, which is still a persistent-compatible version (meaning , no data migration is needed upgrading from v6 to v7).

With the potential path forward of this feature, it is unlikely to be compatible with v7 as of now. I would suggest it should be in V8 with proper feature bit. Also decisions regarding how V6 can upgrade to V8 should be discussed.

@szmyd

szmyd commented Sep 14, 2026

Copy link
Copy Markdown
Collaborator

Also decisions regarding how V6 can upgrade to V8 should be discussed.

It's my understanding that we only use v3 (NuBlox 1.x/2.0) and v7 (NuObject) at the moment. We're currently working on v8 for NuBlox 2.1; so where does v6 come into play?

@xiaoxichen

Copy link
Copy Markdown
Collaborator

@szmyd sorry you are right, I mess up v7 vs v6.. Yes I am talking about v7 ->v8 compatibility

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Production cleanup is not wired into chunk release, and the build and test lifecycle contain blocking defects.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Introduces chunk-level WBC eviction primitives intended to prevent stale cache entries when index chunks are reused.

Changes:

  • Adds allocated-block enumeration and WBC chunk eviction APIs.
  • Adds a deterministic chunk-reuse regression test.
  • Updates test helpers, build configuration, and package version.
File summaries
File Description
conanfile.py Bumps version to 8.2.1.
src/CMakeLists.txt Enables prerelease compilation.
src/include/homestore/index/wb_cache_base.hpp Exposes chunk eviction API.
src/lib/device/chunk.cpp Enumerates allocated blocks.
src/lib/device/chunk.h Declares block enumeration.
src/lib/index/wb_cache.cpp Implements chunk-wide eviction.
src/lib/index/wb_cache.hpp Declares eviction override.
src/tests/CMakeLists.txt Registers the regression test.
src/tests/btree_helpers/btree_test_helper.hpp Makes flip arguments explicit arrays.
src/tests/test_common/homestore_test_common.hpp Makes flip arguments explicit arrays.
src/tests/test_index_chunk_selector.cpp Adds the chunk-reuse regression test.
Review details
  • Files reviewed: 11/11 changed files
  • Comments generated: 4
  • Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/CMakeLists.txt
Comment on lines 68 to +69
target_link_libraries(homestore ${COMMON_DEPS})
target_compile_definitions(homestore PRIVATE _PRERELEASE)
Comment thread src/tests/CMakeLists.txt
Comment thread src/lib/index/wb_cache.cpp
Comment thread src/tests/test_index_chunk_selector.cpp
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants