Stable 25.8: Bump to 25.8.30

[zvonand]

Changelog category (leave one):

• Not for changelog (changelog entry is not required)

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Performance tests
• Aarch64 tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• OAuth (5m)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [76aaa04]

[zvonand]

@blau-ai

[blau-ai]

CI triage — 3 red checks: 1 real & PR-related (trivial fix), 1 policy (DCO), 1 aggregate

Head SHA dc287f37. Everything else is green (Fast test, all other Stateless splits, Stress, all Regression suites). Verdict:

Check	Class	PR-caused?
Stateless tests (amd_binary, old analyzer, s3 storage, DatabaseReplicated, sequential)	Real test failure	Yes — trivial fix below
DCO	Sign-off policy	Not a code issue — needs a human decision
PR	Aggregate gate	Just reflects the two above

---

1. Stateless / DatabaseReplicated — 04296_alter_rename_column_in_memory_rollback (PR-caused, 1-line fix)

Only 1 test failed in the whole suite (575 passed):

04296_alter_rename_column_in_memory_rollback   FAIL   return code: 47
Code: 47. DB::Exception: Missing columns: 'd' while processing:
'SELECT id, d FROM t_rename_rollback ORDER BY id ASC' ... (UNKNOWN_IDENTIFIER)

Root cause — a missing tag on a newly-added test, not a product bug. This test is new to this PR; it was added by backport commit 17ca607 ("Backport ClickHouse#104822 … Fix race between ALTER RENAME COLUMN and merge causing data loss"). It injects the mt_alter_throw_in_start_mutation failpoint into a plain MergeTree table, runs ALTER … RENAME COLUMN, and asserts that after the throw the in-memory metadata was reverted so SELECT d still works. That invariant is specific to the standalone StorageMergeTree::alter code path. Under --database-replicated the RENAME is driven through the replicated DDL/metadata layer instead, so the storage-level in-memory revert isn't the source of truth and SELECT d legitimately fails → the test is simply not applicable in that mode.

The same backport already knows this: every sibling failpoint/rollback test it adds is tagged no-replicated-database, and the DatabaseReplicated case has its own dedicated test (04311). 04296 is the only one that got no-parallel but not no-replicated-database:

04296_alter_rename_column_in_memory_rollback          # Tags: no-parallel                          <-- missing
04308_alter_rename_column_prepared_mutation_cleanup   # Tags: no-parallel, no-replicated-database
04309_alter_rename_column_post_registration_rollback  # Tags: no-parallel, no-replicated-database
04310_alter_rename_column_settings_rollback           # Tags: no-parallel, no-replicated-database
04311_alter_rename_column_replicated_db_rollback      # Tags: no-parallel, zookeeper, no-replicated-database
04312_alter_rename_column_durable_rollback_throw      # Tags: no-parallel, no-replicated-database

This is deterministic (a failpoint, not a race) and confined to the DatabaseReplicated split — the test passed in every other sequential suite (debug, AsyncInsert, ParallelReplicas, distributed-plan). Not flaky, not infra.

Proposed fix (tests/queries/0_stateless/04296_alter_rename_column_in_memory_rollback.sh), matching its siblings:

 #!/usr/bin/env bash
-# Tags: no-parallel
+# Tags: no-parallel, no-replicated-database
 # Tag no-parallel: uses fail points which affect the whole server.

2. DCO (not PR-code-caused — maintainer decision)

There are 87 commits incorrectly signed off … Sign-off not found

These are the upstream v25.8.30 tag commits pulled in by the merge (e.g. ea13bcb, the backport commits, etc.) — they don't carry Signed-off-by trailers. This is the normal state for a stable-bump PR that merges an upstream tag, and is not something to "fix" in code: rewriting sign-offs across 87 commits would rewrite the merged history. This one is a human call — either override/waive the DCO gate for this bump PR (as is customary here) or configure it to skip merge-bump PRs. I won't rewrite those commits.

3. PR (aggregate)

Green once #1 is fixed and DCO is waived — it carries no independent error.

---

Health check

The bump is in good shape: Fast test (6940 passed), Stress, all Regression suites (iceberg/parquet), and every other Stateless split are green. The only functional red is a single mis-tagged new test — a one-line change. The remaining red (DCO) is a sign-off-policy artifact of merging the upstream tag, not a defect.

Delivery: you asked for the fix but didn't say how you'd like it delivered, so I haven't pushed. Want me to commit the one-line tag fix directly to bump/stable-25.8/25.8.30, or open a separate blau/* PR against it? Say the word and I'll do it.

_{— @blau-ai · evidence: praktika result_pr.json for dc287f37, backport commit 17ca607}

[zvonand]

@blau-ai

[blau-ai]

CI triage for #2225 (bump to 25.8.30)

Verdict: 4 red checks, 0 product regressions. 1 is a PR-related test-stability issue (no product impact), 2 are inherent-to-a-bump / infra, and the PR row is just the rollup gate.

Check	Class	PR-caused?
Integration tests (amd_binary, 4/5)	new test fails deterministically, but server never crashes	Introduced by this PR (new backported test), not a product regression
DCO	87 upstream backport commits without Signed-off-by	No — inherent to a version bump
GrypeScanServer (-alpine) / Grype Scan	1 HIGH CVE in the alpine image only	No — base-image / scanner, not this diff
PR	aggregate rollup of the above	—

Everything else is green: all builds, Fast test (6940 passed), stateless/AST-fuzzer/BuzzHouse, the other 4 integration shards (1/5, 2/5, 3/5, 5/5 — their broken: N counts are allow-listed known-broken tests, not failures), non-alpine Grype (server 0 vulns, keeper 0, keeper-alpine 0), and release regression.

---

1. Integration tests (amd_binary, 4/5) — PR-related test, but no crash / no regression

Only one real failure in the shard (fail: 1, passed: 885; the broken: 1 is an allow-listed test, non-blocking):

test_storage_s3_queue/test_parallel_inserts.py::test_batch_set_processing_failure_does_not_crash
E   RuntimeError: 1 did not match expectations in 120 retries
test_storage_s3_queue/test_parallel_inserts.py:350

What it is. This test is new in this PR — it arrives with the straight upstream backport 6b0c9630d0f "Backport ClickHouse#108977 to 25.8: Fix out-of-bounds crash in ObjectStorageQueueSource::FileIterator::next" (not on the base stable-25.8). The backport is complete: the fix (compact_file_metadatas guard in ObjectStorageQueueSource.cpp:281-285), the profile event (ProfileEvents.cpp:868), and the failpoint (FailPoint.cpp:68) are all present and wired.

Why it's not a regression. The test guards against a server abort. In the log the server stays alive the whole time (SELECT 1 succeeds on every one of the 120 retries) — the OOB crash the fix targets does not happen. What times out is the secondary assertion that the failing-batch path was actually exercised, i.e. that the ObjectStorageQueueFailedToBatchSetProcessing profile event increments above baseline. It never does within the 120 s window.

Why the assertion is fragile. The test has to make two things land in the same listing batch: (1) a file made non-processable via the object_storage_queue_skip_one_file_in_batch failpoint, which is a ONCE failpoint (FailPoint.cpp:68), and (2) a pre-created keeper /processing node so the batch's tryMulti fails (ObjectStorageQueueSource.cpp:250,271). If the streaming task's first batch attempt doesn't align the one-shot skip with the conflict file, the ONCE failpoint is consumed and the event can never fire — so the timeout is deterministic, not random. It failed identically on both the initial run (run-id=0) and the automatic retry (run-id=1), which rules out plain flakiness and points at a fragile reproduction harness rather than a code defect.

Suggested fix. Since the product code is correct (no crash), don't block the bump on the test's reproduction race. Two options:

• Recommended — quarantine via the existing tests/broken_tests.yaml (already used for several test_storage_s3_queue cases marked "KNOWN - Unstable upstream"), and track a stabilization upstream:

  --- a/tests/broken_tests.yaml
  +++ b/tests/broken_tests.yaml
  +- name: test_storage_s3_queue/test_parallel_inserts.py::test_batch_set_processing_failure_does_not_crash
  +  reason: 'INVESTIGATE - reproduction race (ONCE failpoint + keeper multi must land in one batch); server does not crash, only the profile-event assertion times out'

• Or stabilize the test itself (upstream-worthy): make the skip-one-file failpoint retriggerable / loop the failpoint+MV setup until the event increments before starting the wait, instead of relying on a single one-shot batch. That's a change to an upstream test, so it should ideally go upstream first.

I'd lean on the quarantine for the release bump. Happy to prepare it — see the delivery note at the bottom.

2. DCO — expected for a bump, not actionable by me

DCO reports 87 commits "incorrectly signed off" — e.g. ea13bcb, b0f124a, 54b5643 … all Sign-off not found. These are the upstream ClickHouse backport commits (robot-clickhouse, upstream maintainers) that this PR merges in; they legitimately lack an Altinity Signed-off-by trailer. This is the normal state of every upstream-bump PR and is not something introduced by the bump work itself. It can't be fixed without rewriting 87 upstream commit messages (history rewrite), which I won't do. Recommendation: handle as usual for bumps — admin override / merge with the DCO check waived.

3. GrypeScanServer (-alpine) — base-image finding, not this diff

The alpine server image scan fails with 1 HIGH: CVE-2026-14456 (severity High, matched via nvd:cpe). Note it is only the -alpine server image — the regular server image, keeper, and keeper-alpine (same ClickHouse 25.8.30 build) all scan clean (0 high/critical). Because the ClickHouse binary is identical across those images, the finding is a property of the alpine base image / an NVD CPE match, not of anything this PR changed in the source. The published Grype report is HTML-only (no package/fixed-in column), so the exact apk package isn't in the artifact. Recommendation: treat as an image/scanner issue — refresh the alpine base image / package, or add a scanner allow-list entry if it's a CPE false-positive; there's no ClickHouse source change in this PR that can address it.

---

Delivery

The mention didn't specify how you want the fix delivered, so I haven't pushed anything. The only PR-related item worth a code change is the broken_tests.yaml quarantine above. Tell me which you'd like and I'll do it:

• (a) open a separate PR from a blau/* branch against bump/stable-25.8/25.8.30, or
• (b) commit the one-line quarantine directly to bump/stable-25.8/25.8.30.

If instead you'd rather I try to stabilize the test itself (option under §1) or dig further into the alpine CVE package, say the word.

Evidence: praktika result_pr.json for ef0f43d, per-test logs integration_run_test_storage_s3_queue_test_parallel_inserts_py_0/1.log, DCO check-run summary, and the alpine vs non-alpine Grype reports. I can't build/run ClickHouse here, so the OOB-fix correctness and the quarantine will ultimately be confirmed by CI on whichever branch we land it.

[zvonand]

@blau-ai the same test test_batch_set_processing_failure_does_not_crash seems to be OK in a sibling PR: #2224

Investigate this

[blau-ai]

Investigated. Short version: this is a flaky reproduction timeout in a newly-added timing-sensitive test, not a real regression, and nothing about the stable-25.8 bump makes it behave differently from #2224. The code the test exercises is byte-identical on both branches.

What actually failed (it did not crash)

The one failing test on Integration tests (amd_binary, 4/5) is test_storage_s3_queue/test_parallel_inserts.py::test_batch_set_processing_failure_does_not_crash. It failed on both the initial run and the retry, but with a benign error — the server never aborted:

>   run_with_retry(lambda x: x > failures_before, batch_set_processing_failures, retries=120)
E   RuntimeError: 1 did not match expectations in 120 retries
test_parallel_inserts.py:350  ->  :49

Throughout all 120 retries the SELECT 1 health-check kept succeeding, so the server stayed alive the whole time (which is exactly what the fix guarantees). What did not happen is the precondition: the ObjectStorageQueueFailedToBatchSetProcessing profile event never incremented past its snapshot, i.e. the test could not reproduce the racy "both conditions in one batch" scenario within the 120s window. It failed while setting up the crash, not while surviving it.

That reproduction is inherently probabilistic: it needs the failpoint-skipped file and the pre-created keeper conflict node to land in the same listing batch, with enable_hash_ring_filtering=1 and s3queue_processing_threads_num=1. Whether that alignment happens within the window is timing-dependent.

Why 2224 is green and 2225 is red — same code, different luck

Both PRs are the 25.8.30 bump and both contain the same commit that added this test together with its fix:

• 6b0c9630d0f — Backport Fix out-of-bounds crash in ObjectStorageQueueSource::FileIterator::next ClickHouse/ClickHouse#108977 to 25.8: Fix out-of-bounds crash in ObjectStorageQueueSource::FileIterator::next — is present on both bump/stable-25.8/25.8.30 (this PR) and bump/antalya-25.8/25.8.30 (Antalya 25.8: bump to 25.8.30 #2224).

Diffing the entire engine directory between the two PR heads:

git diff origin/bump/antalya-25.8/25.8.30 origin/bump/stable-25.8/25.8.30 -- src/Storages/ObjectStorageQueue/
 StorageObjectStorageQueue.cpp | 8 ++++----
 StorageObjectStorageQueue.h   | 2 +-
 registerQueueStorage.cpp      | 2 +-

Those are the only differences, and every hunk is cosmetic API adaptation (configuration->getFormat() → configuration->format, and the initialize(...) signature) — antalya-vs-stable base-API noise. ObjectStorageQueueSource.cpp, ObjectStorageQueueIFileMetadata.cpp and FailPoint.cpp — the batch-set-processing, hash-ring filtering, keeper-multi and compaction code this test actually drives — are identical on both branches.

So the pass/fail difference cannot come from a code difference in the tested path. It is the same probabilistic reproduction resolving favorably on 2224 and unfavorably on this run of 2225. (For what it's worth, 2224's integration jobs are currently re-running / pending, so its own result is being regenerated too.)

Recommendation

• Re-run the failed Integration tests (amd_binary, 4/5) job. The failure is a flaky setup timeout, not a crash or a regression, and it's safe to retry — it doesn't block the bump.
• Longer term this is a test-robustness issue that belongs upstream with Fix out-of-bounds crash in ObjectStorageQueueSource::FileIterator::next ClickHouse/ClickHouse#108977 (e.g. make the "both conditions in one batch" scenario deterministic, or raise the retry budget), not something to fix in this bump PR. Happy to open a follow-up hardening PR for the test if you'd like it.

Evidence: failing test log .../integration_tests_amd_binary_4_5/integration_run_test_storage_s3_queue_test_parallel_inserts_py_0.log (and _py_1.log for the retry) under PRs/2225/ef0f43d50507d0d461cbbd9e71caceec9dfb1980/.

[zvonand]

@blau-ai maybe mark this test_batch_set_processing_failure_does_not_crash as flaky?

[blau-ai]

@zvonand Confirmed it's flaky, and I agree it should be marked. Here's the evidence and a proposed change.

What's failing

Only one test fails on this PR, in Integration tests (amd_binary, 4/5) (fail: 1, passed: 885); it was even auto-retried:

test_storage_s3_queue/test_parallel_inserts.py::test_batch_set_processing_failure_does_not_crash — FAIL

>   run_with_retry(lambda x: x > failures_before, batch_set_processing_failures, retries=120)
test_storage_s3_queue/test_parallel_inserts.py:350
E   RuntimeError: 1 did not match expectations in 120 retries

Why it's flaky (not a real regression)

The test waits for the ObjectStorageQueueFailedToBatchSetProcessing profile event to increment — i.e. for the engine to actually attempt the batch that hits both crash conditions. On this run that batch simply wasn't attempted within the 120s window, so the wait timed out. Throughout the wait SELECT 1 kept succeeding, so the server never crashed — the reproduction is just timing-dependent (it depends on the streaming task scheduling that specific batch against the pre-created keeper conflict node). This test was added by us in the ClickHouse#108977 backport (6b0c9630d0f).

Proposed change

Mechanism used here for flaky integration tests is tests/broken_tests.yaml (already lists many test_storage_s3_queue/* cases). I'd add an entry with a message filter so only this timeout is treated as broken — a genuine crash regression would abort the server and surface a different error, and would still fail the job (which is the whole point of a ..._does_not_crash test):

--- a/tests/broken_tests.yaml
+++ b/tests/broken_tests.yaml
@@ -116,6 +116,9 @@
 - name: test_storage_s3_queue/test_4.py::test_list_and_delete_race
   reason: 'KNOWN - Unstable upstream'
   message: AssertionError
+- name: test_storage_s3_queue/test_parallel_inserts.py::test_batch_set_processing_failure_does_not_crash
+  reason: 'KNOWN - Flaky: crash-repro batch not always scheduled within the retry window'
+  message: did not match expectations in 120 retries
 - name: test_storage_s3_queue/test_5.py::test_failed_startup
   reason: 'INVESTIGATE: Unstable on tsan'
   check_types:

The message is specific to this test's retries=120 wait, so it won't mask an actual out-of-bounds crash (that path fails differently). If you'd rather mark it unconditionally, I can drop the message line.

How would you like it delivered?

Per my push policy I don't push without an explicit choice, so tell me which you want:

1. Commit directly to this PR branch (bump/stable-25.8/25.8.30), or
2. Open a separate PR from a blau/* branch targeting stable-25.8.

Reply with 1 or 2 (or "no message" if you also want the unconditional variant) and I'll apply it.

[zvonand]

@blau-ai into this branch

[blau-ai]

Done — committed directly to this branch (bump/stable-25.8/25.8.30) as you asked.

Commit: 76aaa041e70 — Mark test_batch_set_processing_failure_does_not_crash as flaky

--- a/tests/broken_tests.yaml
+++ b/tests/broken_tests.yaml
@@ -116,6 +116,9 @@
 - name: test_storage_s3_queue/test_4.py::test_list_and_delete_race
   reason: 'KNOWN - Unstable upstream'
   message: AssertionError
+- name: test_storage_s3_queue/test_parallel_inserts.py::test_batch_set_processing_failure_does_not_crash
+  reason: 'KNOWN - Flaky: crash-repro batch not always scheduled within the retry window'
+  message: did not match expectations in 120 retries
 - name: test_storage_s3_queue/test_5.py::test_failed_startup
   reason: 'INVESTIGATE: Unstable on tsan'
   check_types:

The message: did not match expectations in 120 retries filter scopes this to the timing-dependent setup timeout only — a genuine out-of-bounds crash aborts the server and fails differently, so it will still fail the job (which is the whole point of a ..._does_not_crash test).

CI will re-run on the new head; the previously-failed Integration tests (amd_binary, 4/5) job should no longer be blocked by this case.