Update every dependency, reuse doublets 0.5.0, open every layer for extension, and prove cross-language parity - #101
Merged
Merged
Conversation
Adding .gitkeep for PR creation (default mode). This file will be removed when the task is complete. Issue: #100
…ators The Rust CLI kept its own ad-hoc write path, so four behaviours the C# CLI gets for free from `DecorateWithAutomaticUniquenessAndUsagesResolution` were missing: deleting a link left dangling references behind, the cascade did not chain, an update into an existing pair created a duplicate, and the named variant of the cascade delete diverged too. `LinkStorage` now implements the upstream `Links`/`Doublets` traits (an `each_core` port that reproduces the index semantics of the unit store, including that a link with a null part is not reachable through a `(source, target)` lookup), which lets the same upstream resolver stack C# uses sit on top of it. Two layers had to follow: - The transactions decorator recorded a single transition per logical write, so every link a cascade touched was invisible to the log and rollback and branch switching silently lost those changes. It now folds the observed `(before, after)` callbacks the way C#'s `TransactionsDecorator.RunWrite` does — first `before`, last `after`, first-seen order — and derives the transition kind per link, because a cascade can delete a link during an `update` and recording that as an `Update` makes the revert a no-op. - The query processor gained C#'s `intendedFinalStates` / `RestoreUnexpectedLinkDeletions` pass, so a link a merge removed as a side effect is put back instead of leaving the query half-applied. The two `get_link(..).unwrap()` sites that assumed the link survived the write are guarded rather than panicking. A cross-language harness lives in `docs/case-studies/issue-100/evidence/cli-parity/`: 21 of 22 scenarios now produce byte-identical databases *and* accept the same queries. The remaining one is an upstream C# defect, reproduced in isolation in `evidence/csharp-merge-usages/`: `ILinksExtensions.MergeUsages` builds its substitutions with the two-argument `Link<TLinkAddress>` constructor, which `SetValues` reads as `(index, source)` with a null target rather than as `(source, target)`, so every usage it repoints has its source overwritten and its target blanked. It is recorded as a known difference that turns red if the two languages ever agree.
…e constants A names database stores external references as Hybrid<uint> values — 0u32.wrapping_sub(address) — but LinkStorage reported the internal-only LinksConstants, where those values either land inside internal_range or sit exactly on a service constant: the external reference of link 4 is u32::MAX - 3, which the internal-only constants define as `any`. Two independent defects fell out of that: * LinkStorage::get_or_create called Doublets::search instead of its own inherent search. LinkStorage implements Doublets for &mut LinkStorage so a borrowed store can be decorated, and inside an inherent &mut self method the receiver's type is exactly &mut LinkStorage, which method resolution reaches before it derefs to the inherent impl. The trait search treats `any` as a wildcard, so naming a link 'UnicodeSymbol' silently reused the pinned type's own name pair. * UnicodeStringStorage read the first name holder out of a HashMap-backed query, so a name shared with a pinned type resolved differently depending on hash order. Report the hybrid constants, fully qualify the inherent calls, and order name holders by address. Adds regression tests for both.
PersistentTransformationDecorator applies stored `Always` and `Once` triggers after every write, mirroring the C# decorator down to the on-disk schema — (Always ((Condition <text>) (Substitution <text>))) — so the two implementations can read each other's trigger databases. Triggers live in a <database>.triggers.links sidecar by default, or in the decorated database itself. NamedTypeLinks is not object safe, so the trigger store is driven by free generic helpers plus a small dispatch macro instead of a trait object. create/ensure_created/get_or_create are infallible in that trait, so a failure raised while applying triggers is parked in pending_error and surfaced by the next update, delete or save.
Member
Author
|
I don't see changes in C#, everything there must be also updated to the latest best practices, latest versions of packages and so on. |
The trigger decorator was ported in the previous commit but nothing reached it: `--always`, `--once`, `--never`, `--triggers`, `--triggers-file` and `--embed-triggers` were accepted by the C# tool only, so the two CLIs did not offer the same feature set. `--always`/`--once`/`--never` store or remove a trigger, `--triggers` lists them, and `--triggers-file`/`--embed-triggers` decide where they live, exactly as `Program.cs` resolves them: a sidecar `<db>.triggers.links` by default, the main database when embedding is asked for.
Which address a new link gets is observable -- it is the identifier the query answers with, and the one the next query has to use -- so the two implementations have to agree on it. They did not. `ResizableDirectMemoryLinks` keeps an allocation counter plus a free list threaded through the freed links, so it reuses a freed address before growing, most recently freed first, and shrinks when the link at the top is deleted. `LinkStorage` only ever incremented a counter, so after any deletion the two stores drifted apart, and `(() ((1 2)))` left a dangling `(3: 1 2)` behind where C# produced `(1: 1 2)`. `LinkStorage` now keeps the same allocation counter and free list, and `ensure_created` gives back the addresses it passed over on the way to the requested one, the way `ILinksExtensions.EnsureCreated` does. The free list is persisted as a `# unused:` comment line so that the order survives between the one-shot CLI invocations that a scenario is made of; a database written without that line still loads, with the list reconstructed from the addresses the file skips.
`--changes` is a contract: it says what a query did, and callers diff those lines. The Rust CLI answered differently from the C# one in three ways. An auto-created reference was reported as `() ((id: id id))`, as if it had appeared from nothing. C# creates the placeholder and then updates it, so the update is reported against the placeholder it started from -- `((id: 0 0)) ((id: id id))`. `validate_links_exist_or_will_be_created` now returns the before/after pair of each reference it created rather than only the finished link. A delete was reported as the one link that was asked for, while C# reports the cascade: deleting a link deletes every link that still used it, and `AdvancedMixedQueryProcessor.RemoveLinks` passes a handler down so each of those removals is reported too. `NamedTypeLinks` gains `delete_observed`, the trait-level equivalent of that handler, which the query processor uses wherever it deletes. Finally, the order of the reported changes was not reproducible. `simplify_changes` enumerated `HashSet`s and a `HashMap`, whose order depends on a per-process random seed, where C# enumerates a `HashSet` and a `GroupBy` in insertion order. Both are now iterated in first- occurrence order, so the same query prints the same lines every run.
Unseal the decorators, mark their public members virtual, and give the disposable ones the protected virtual Dispose(bool) pattern so a subclass can release resources of its own. Publish the trigger query record and the internal name prefix, which the Rust library already exposes. ExtensibilityTests subclasses four of the decorators and asserts the seam reflectively, so re-sealing a class or dropping a virtual fails the suite.
Two small programs, one per language, print the constants a store with external-reference support reports. platform-data 2.0.0 starts the external range on `continue`, so `is_external(continue)` is true; the C# original starts it one address later and answers False. The Rust program exits non-zero once upstream fixes the overlap.
An unbound substitution variable -- or a `*` in a substitution -- is unspecified, not an address. C# marks it with `links.Constants.Any`, a value the store understands: a create writes null there, a lookup runs through `Each` and reads it as a wildcard, and an update keeps the half already stored. The Rust processor marked the same thing with `u32::MAX`, which the store underneath does not recognise (its `any` is `2147483644`), so `() (($a $a))` stored the literal `4294967295` in both halves where C# stores `(1: 0 0)`. Resolve at the write boundary instead, so `u32::MAX` stays this crate's single internal marker and restriction matching is untouched. Five of six probe shapes diverged before the fix; all six agree after it. Covered by six new parity tests and six new harness scenarios.
Restates the six asks in the issue, records what each one found, and attributes the one remaining C#/Rust divergence to the upstream MergeUsages defect rather than to this repository. Also removes the root .gitkeep placeholder, which the repository layout test rejects, and moves the verification logs under dev/log per the convention issue 96 established.
konard
changed the title
CI's file-size gate (csharp/scripts/check-file-size.mjs, 1000 lines max) failed on this branch: query_processor.rs had grown to 1110 lines while resolving unspecified substitution halves, and transactions/mod.rs to 1074 while recording cascaded transitions. Both files are split the way the repository already splits large ones: a child module that can reach the parent's private items, so nothing had to be made more visible than pub(super). rust/src/query_processor/mutations.rs -- the write side (delete, create/update, restore of unexpected deletions, EnsureLinkCreated), mirroring the C# AdvancedMixedQueryProcessor.Mutations.cs split, next to the existing query_processor/matching.rs. rust/src/transactions/recovery.rs -- the paths that read the sidecar log back: applying and reverting a single transition, crash recovery and log retention. Pure code motion: 239 Rust tests, cargo fmt, clippy -D warnings and the CLI parity harness (39 PASS + 1 KNOWN) are unchanged.
Member
Author
Both parts are covered — here is the evidence. C# code changes. Every decorator is unsealed, its public members are That is the C# half of ask #4 ("all abstractions with all trust for extension, as much public members as possible") and ask #5 ("a library itself to simplify alternative/custom CLIs"). The rest of the C# work in this PR is the reference side of ask #6: the C# CLI is the parity oracle that the Rust fixes are measured against, so where the two disagreed the change landed in Rust. The five divergences and the one upstream exemption are tabulated in the PR description and in the case study. Package versions. Every C# dependency was checked against nuget.org and every one is already on its newest stable release, so there is nothing to bump — re-verified just now against
The only dependency in the whole repository that was behind was Rust's
Verification: If there is a specific C# best practice you want applied that this misses, name it and I will add it. |
konard
marked this pull request as ready for review
August 29, 2026 19:28
Member
Author
🤖 Solution Draft LogThis log file contains the complete execution trace of the AI solution draft process. 💰 Cost: $72.802135📊 Context and tokens usage:Claude Opus 5: (18 sub-sessions)
Total: (58.2K new + 1.7M cache writes + 74.7M cache reads) input tokens, 744.7K output tokens, $72.802135 cost 🤖 Models used:
📎 Log file uploaded as Gist (24970KB)Now working session is ended, feel free to review and add any feedback on the solution draft. |
Member
Author
✅ Ready to mergeThis pull request is now ready to be merged:
Monitored by hive-mind with --auto-restart-until-mergeable flag |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #100.
Issue #100 is six sentences, and each one is a separate ask: report what is missing in
doublets-rs, update every dependency in every language, lean on the newestdoubletsso less is duplicated here, open every layer up for extension, ship a library and not only a CLI in every language, and make sure the languages actually have the same features.The last ask is the one that generated most of the work. "Nothing is missing in any of languages" cannot be answered by reading code — the two implementations have different stores underneath them — so it is answered by running both CLIs over the same query sequences and diffing the resulting databases. That harness found five real divergences, all fixed here, and one that belongs upstream.
The full write-up, with the root-cause analysis for each, is
docs/case-studies/issue-100/README.md.What changed
1. Reported upstream
Platform.Data.Doublets.Sequences— sequences, Unicode strings, walkers. This repository ports that path by hand, which is exactly the duplication the issue asks us to remove, and it cannot go until upstream has the layer.UInt64LinksTransactionsLayer, Rust has nothing, solink-climaintains its own.MergeUsageswrites null targets and wrong sources — the cause of the one remaining C#/Rust divergence.LinksConstants::external()overlaps the external range withcontinue, sois_external(continue)istruein Rust andFalsein C#.A fifth candidate was investigated and deliberately not filed:
doublets0.5.0'sunit::Storealready implements the exact address-allocation contract this repository reverse-engineered from C# (UnusedLinks+header.first_free,attach_as_firstfor LIFO reuse, tail shrink on delete). There was nothing to ask for — the bug was on our side.Both upstream reproductions are committed as runnable harnesses that exit 0 while the defect reproduces and non-zero once it is fixed, so the day the upstream release lands we find out instead of the workaround quietly outliving its reason.
2. Dependencies, all languages
Every dependency in Rust, C# and JS was checked against its registry. Exactly one was behind:
doubletsThe other 22 were already at their newest published stable version. They are listed at their current versions in §7 of the case study so the check is auditable rather than implied. One deserves a note:
System.CommandLine3.0.0 exists on NuGet but only as a prerelease, so 2.0.11 stays.links-notationremains deliberately aligned at 0.16.1 across Rust and C#, the invariant established in #99.3. Reusing doublets 0.5.0 instead of duplicating it
Before this PR the Rust
LinkStorageresolved uniqueness and cascading deletes with its own code, while C# got both fromILinksExtensions.DecorateWithAutomaticUniquenessAndUsagesResolution.doublets0.5.0 exposesdoublets::decorators, which is the same stack.DoubletsStorage::map_storecomposes any upstream (or caller-written) decorator onto an open database while keeping its path, its advisory lock and its change-detection fingerprint.DoubletsStorage::with_automatic_uniqueness_and_usages_resolutionapplies the C# stack by name.doubletscrate — includingdecorators— is re-exported fromlink_cli, so a downstream crate can build its own stack without a direct dependency that could drift to an incompatible semver.Routing the CLI's
LinkStoragethrough those resolvers is what made a Rust delete cascade into the links that referenced the deleted one, and an update that would duplicate an existing link merge into it, the way C# already did.4. Everything open for extension
link_cliis public, along with the query patterns, the resolved links, the link reference validator and the transition wire-format constants.NamedTypeLinksis documented as the seam: every layer is written against it and every decorator both implements it and wraps another implementation of it, so a cache, an access check or a remote store slots in anywhere — including underQueryProcessor, which never learns what is beneath it.NamedTypesDecorator,NamedLinksDecorator,SimpleLinksDecorator,PinnedTypesDecorator,TransactionsDecorator,VersionControlDecorator,PersistentTransformationDecorator) is unsealed with overridable members; the disposable ones followprotected virtual void Dispose(bool)so a subclass can release resources of its own;PersistentTransformationQueryandInternalNamePrefixare now public.ExtensibilityTestssubclasses four of them and asserts the seam reflectively, so re-sealing a class or dropping avirtualfails the suite instead of silently narrowing the API.The C# library was already a separate project that the CLI consumes, so "a library, not only a CLI" was already true there; what was missing was the extension surface. It is now true in Rust down to the last module. The JS surface is a web front end over the Rust WASM build rather than a third implementation of the CLI, so it inherits whatever
clink-wasmexposes.5. Same features in every language — proved, not asserted
docs/case-studies/issue-100/evidence/cli-parity/run.shruns the same query sequence through both binaries and compares two things rather than one: the final database dump, and one accepted/rejected verdict per query. The verdicts matter because a query both CLIs refuse leaves two empty databases, which a dump-only comparison would happily call a match. The exit status is compared rather than the message text — the two implementations are expected to agree on what they accept, not on how they word a rejection.39 scenarios. Five divergences were found and fixed:
--changesreported different changesHashMapiteration, so it varied with the process's hash seed.LinkStoragedid not report the hybridLinksConstants, and its inherentget_or_createresolved through theDoubletsimpl, which readsanyas a wildcard.Plus one missing feature: the Rust CLI had no persistent transformation triggers. It now has
--always,--once,--never,--triggers,--triggers-fileand--embed-triggers, with the same<database>.triggers.linkssidecar schema — triggers written by either implementation are readable by the other.The unspecified-half bug
A substitution half that no restriction ever bound — a never-bound variable, or a
*— is unspecified, not an address. C# marks it withlinks.Constants.Any, which is a value the store understands, and the store gives it three meanings depending on where it lands: a wildcard inSearchOrDefault, "keep the half already stored" in anUpdate, and null in a create. That is one rule: unspecified → the existing value, or null when there is none.The Rust processor marked the same thing with
u32::MAX, a value its store does not recognise — the store'sanyis2147483644. So:Five of six probe shapes diverged. The fix resolves at the write boundary (
QueryProcessor::resolve_unspecified/search_unspecified) rather than changing the sentinel, sou32::MAXstays the crate's single internal marker and restriction matching — andNamedTypeLinks::search, which is deliberately literal because it backs uniqueness resolution — is untouched.6. The one difference that is not ours
C# leaves
(2: 2 0); Rust leaves(2: 2 2).Platform.Data.Doublets.Link<T>has no two-argument(source, target)constructor — it has(params T[] values)— sonew Link<uint>(a, b)binds toparams, andSetValuesreads a two-element list as(index, source)withtarget = default.MergeUsagesbuilds its replacement links that way, so it repoints usages onto a link with a null target and a source that is really the index.docs/case-studies/issue-100/evidence/csharp-merge-usages/run.shreproduces it againstPlatform.Data.Doubletsdirectly, with nolink-clicode involved. It is recorded in the parity harness as its singleknown_difference— andknown_differencecounts agreement as a failure, so the exemption cannot outlive the upstream fix.How to reproduce the original problems
Tests
cargo fmt --all -- --check(both workspaces)cargo clippy --all-targets --all-features -- -D warningscargo testdotnet format --verify-no-changesdotnet build --configuration Releasedotnet testnode --testevidence/cli-parity/run.shThe runs behind those numbers ship with the PR, in
dev/log/issues/100/pulls/101/verification/, per the convention #97 established.Automated coverage added for each fix: 6 parity tests for the unspecified halves, 6 harness scenarios for address allocation, 8 for triggers, plus the Rust trigger CLI end-to-end tests and the C#
ExtensibilityTeststhat assert the extension seam reflectively.Release
Rust changelog fragments in
rust/changelog.d/(one per change,minorfor the new capabilities,patchfor the parity fixes) and a C# changeset incsharp/.changeset/(minor), so the release workflows pick the version up on merge.Follow-ups
MergeUsagesexemption disappears when Data.Doublets#515 ships; the harness turns red to tell us.is_external(continue)still differs between the languages (data-rs#18). It is unreachable through the CLI — naming address2147483644makes both CLIs try to allocate ~2 billion links — but a library consumer calling it directly will see it.