Transaction log deletion is database-wide on RocksDB; table-scoped requests are rejected

[cb1kenobi]

Companion to HarperFast/harper#2064 (fixes HarperFast/harper#2049 — RocksDB: table-scoped delete_transaction_logs_before destroys the entire database's transaction log).

Updates reference/database/transaction.md:

• Adds a delete_transaction_logs_before section: on RocksDB (the default engine) deletion is database-wide because all tables in a database share one transaction log; as of v5.2.0 a request naming a table is rejected with a 400 (previously the table scope was silently ignored and the whole database's log was purged), and a nonexistent table returns a 404. On LMDB, table scopes deletion to that table, unchanged.
• Notes on delete_audit_logs_before that it is deprecated in favor of delete_transaction_logs_before, and that on RocksDB it always errors since it requires table.

npm run build and npm run format:check pass locally.

Generated by Claude (Fable 5).

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request updates the database transaction documentation to deprecate the delete_audit_logs_before operation and introduces documentation for the delete_transaction_logs_before operation, detailing its behavior on RocksDB and LMDB. The feedback suggests adding a dedicated ### Transaction Log Operations section header to improve the document's structure and clarity.

[github-actions]

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-618

This preview will update automatically when you push new commits.

[github-actions]

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-618

This preview will update automatically when you push new commits.

[github-actions]

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-618

This preview will update automatically when you push new commits.

[kriszyp]

Probably should clarify the response, since it is returns a job-id.

_{🤖 Reviewed with Codex}

[github-actions]

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-618

This preview will update automatically when you push new commits.

[Ethan-Arrowood]

I went through this against the merged implementation in harper#2064 rather than taking the prose on trust — the 400 guard for a RocksDB table scope, the engine-agnostic 404 that sits before the engine branch, cleanup_deleted_records against Table.deleteHistory, the background-job dispatch, and the response body all check out line by line. Every factual claim in the diff is accurate, and all eight points from the earlier rounds (Gemini's section header, @kriszyp's five Codex findings, Barber AI's two) are correctly resolved in ffe30c44. I'm not re-raising any of those.

What's left is what the page doesn't say, which on a destructive operation is the part that matters. Four gaps:

1. On LMDB, omitting table is a silent no-op — and the section's only example omits it. That request resolves getTable() to undefined, falls into the if (!table) branch, finds no RocksDatabase instance, and returns entries_deleted: 0 with job status COMPLETE. An LMDB operator copying the example verbatim schedules a retention job that quietly does nothing and reports success. Harper's own validator acknowledges this (validation/transactionLogValidator.ts:20: // \table` will need to be required for lmdb, but not for rocksdb`). Inline suggestion below.

2. Requires clustering: Yes in the comparison table is false on v5/RocksDB. The 5.0 release notes describe the transaction log as Harper's "own native transaction log as a write-ahead log (WAL) for RocksDB" — unconditional, not a clustering feature; readTransactionLog just delegates to harperBridge.readAuditLog with no stream involvement; and reference/backups/overview.md says every managed backup includes the transaction log with no clustering caveat. As it stands the row directly contradicts the Storage row above it, which this PR rewrote. A RocksDB reader concludes they have no transaction log unless they've configured replication, then can't reconcile that with a page telling them how to delete it. Inline suggestion below. (The Available since v4.1.0 row has the same v4-era problem, but I'd leave that one for a separate pass.)

3. A database-wide destructive purge is documented in plain body prose. Line 146 is the sentence that tells an operator who wanted to prune one table to instead purge every table's history, and it sits in an unmarked paragraph. The reference tree uses admonitions for exactly this (:::warning Data safety in reference/database/api.md:54). Three things are missing beyond the wrapper: what is actually lost (purgeLogs() deletes whole log files — no per-table survivor, no undo), and the recovery route (a managed backup restores the transaction log alongside the data, per reference/backups/operations.md:113). Inline suggestion below covers both.

   The third piece I couldn't establish myself and it's a genuine question for you or @kriszyp: what happens if you purge below a lagging peer's replication position? purgeLogs looks like a whole-file delete with no watermark check at the bridge layer. If there's no guard, "do not purge below your slowest replica's position" belongs in that admonition. If rocksdb-js refuses to delete unshipped files, that's reassuring and also belongs there. Either answer improves the page — silence is the only bad option, so I left it out of the suggestion rather than guess.

4. No entry in release-notes/v5-lincoln/5.2.md. This is breaking in three ways: table-scoped delete_transaction_logs_before now errors where it previously returned success; delete_audit_logs_before now always errors on the default engine; and a nonexistent table now 404s on LMDB where it previously returned a clean success. Anyone with a scheduled retention job hits one of the first two on upgrade. 5.2.md is actively maintained with prose sections and a reference-page badge isn't a substitute. A short ### Transaction Log Deletion section would do it. Fine as a follow-up if you'd rather not grow this PR — but it shouldn't ship without one.

Everything else below is a suggestion, take or leave. Items 1-3 are cheap enough that I'd land them here.

_{sent with Claude Opus 5}

[cb1kenobi]

@Ethan-Arrowood thanks for the verification-grade pass. All six inline threads are addressed in 5cc6c51 (replies on each), and gap 4 landed in this PR rather than as a follow-up: release-notes/v5-lincoln/5.2.md now has a "Transaction Log Deletion" section covering all three breaking behaviors (RocksDB table-scope rejection, delete_audit_logs_before always failing on RocksDB, nonexistent-table 404 on either engine) with a link to the updated reference page. Your replication-lag question turned out to be answerable — no purge-side guard exists, but the sender forces a bounded base copy when a peer's requested start predates retained history (shouldForceBaseCopyForRetention, harper-pro#277), so the admonition now states the real cost: full resync, not data loss. — Claude (Fable 5)

[github-actions]

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-618

This preview will update automatically when you push new commits.

[kriszyp]

Since we are editing the Audit Log / Transaction Log table... Let's just eliminate it and make it clear there is no such thing as a difference between an "audit log" and a "transaction log". That's an ancient relic of NATS that we don't talk about anymore. We just have a transaction log and that's it.

• reference/database/transaction.md:159: The new parameter documentation says that omitting both database and schema fails validation before a job starts, so “any rejection surfaces when the job runs” is now too broad. A client making that mistake will not receive the job ID described here and cannot poll get_job. Could you distinguish request-validation failures from errors raised after a job has been accepted, for example: “Once accepted, the operation runs as a background job; operation-time failures surface through get_job”?

— KrAIs (Codex)

• release-notes/v5-lincoln/5.2.md:22: “Always fails with the same guidance” conflicts with the next bullet and the operation reference: a nonexistent table fails with a not-found error before the RocksDB guidance is reached. Could you qualify this as an existing RocksDB table, matching reference/database/transaction.md, so upgrade guidance does not promise the wrong error for typo'd table names?

— KrAIs (Codex)

• reference/database/transaction.md:222: The comparison table now correctly says RocksDB transaction logging uses the native WAL and does not require clustering, but this related-doc description still says clustering setup is required for transaction logs generally. Could you qualify it as LMDB/clustering-stream transaction logs, or describe the linked page without claiming clustering is universally required?

— KrAIs (Codex)

_{🤖 Reviewed with Codex}

[cb1kenobi]

@kriszyp — done in 98b1a99. Agreed on the core point: there's one transaction log, and the "audit log" as a separate thing is a NATS-era relic. The table was the main offender, so it's gone, along with the "two complementary mechanisms" framing.

Audit log vs. transaction log

• Deleted the Audit Log / Transaction Log comparison table entirely.
• The page now opens by stating there is a single transaction log per database.
• Added an :::info callout that names the legacy surface explicitly — read_audit_log, delete_audit_logs_before, logging.auditLog, and @table(audit:) are older names that all act on that one log — so a reader who meets those names doesn't infer a second mechanism.
• Merged the Audit Log Operations / Transaction Log Operations split into one ## Operations section (the split itself implied two logs), and reworded the read_audit_log / delete_audit_logs_before prose to say "transaction log".

The three Codex findings

1. transaction.md:159 — reworded to distinguish request-validation failures from operation-time failures. Omitting both database and schema is now described as a synchronous rejection with no job ID (nothing to poll); once accepted, the job runs and operation-time failures (RocksDB table-scope, nonexistent table/database) surface via get_job.
2. 5.2.md:22 — qualified. delete_audit_logs_before now "fails for an existing table with the same guidance," with the nonexistent-table case pointing at the not-found bullet below it, so the note no longer promises the wrong error for a typo'd table name.
3. transaction.md:222 (Related Documentation) — the Replication line no longer claims clustering is required for transaction logs. It now describes replication/clustering as consumers of the log, consistent with the RocksDB-native-WAL messaging in the rest of the page.

npm run build and prettier --check both pass. Thanks for pushing on this — the page is clearer without the false dichotomy.

— Claude (Opus 4.8)

[github-actions]

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-618

This preview will update automatically when you push new commits.

[cb1kenobi]

Ran a Codex review pass over the PR and fixed the findings in a7ad4a0 (no blockers; each verified against the harper / rocksdb-js source):

• Extended the single-log terminology cleanup across sibling pages — the audit-log-vs-transaction-log framing also lived in database/overview.md, database/api.md, operations-api/operations.md, and logging/operations.md. Those now describe one transaction log, so the "no such thing as a difference" message is consistent site-wide, not just on this page.
• Corrected the delete_transaction_logs_before result fields — the docs claimed start_timestamp/end_timestamp on LMDB, but DeleteTransactionLogsBeforeResults is constructed with no args and the LMDB branch only sets entries_deleted, so those never appear. Now documents entries_deleted + log_files_deleted (0 on LMDB).
• Relabeled the get_job example from "a rejected request" to "a failed job" — it's an accepted-then-failed job, keeping the request-validation-vs-job-time distinction clean.
• Softened the RocksDB purge warning — purgeLogs deletes whole log files, so the previous wording overstated entry-level precision.
• Fixed logging.auditLog default in logging/configuration.md (false → true) to match the shipped default (static/defaultConfig.yaml) and this page.

npm run build and prettier --check pass.

— Claude (Opus 4.8)

[github-actions]

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-618

This preview will update automatically when you push new commits.

[kriszyp]

Looks good, one small note about transaction vs audit log terminology.
_{🤖 Reviewed with Codex}

[kriszyp]

The transaction page now explains that the audit log is not a separate mechanism, and on RocksDB changes are stored in the database-wide WAL rather than a corresponding audit table. This touched configuration entry still states that every change is written to an audit table, so readers get conflicting storage models across the two pages. Could you describe this as enabling transaction logging, with read_audit_log retained as the legacy operation name, and avoid promising an audit-table implementation that does not apply to RocksDB?

— KrAIs (Codex)