You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Split out of #26. The relation write path rewrites the whole .archcore/.sync-state.json on every mutation. Before choosing an optimization, add benchmarks so the decision rests on measurements.
Current behavior (v0.8.3)
manifestStore.mutate clones the manifest and calls SaveManifest (internal/mcp/tools/manifest_store.go:87,91). SaveManifest re-marshals the whole file with MarshalIndent (internal/sync/manifest.go:327).
add_relation, remove_relation, and remove_document all go through mutate. archcore status does the same full rewrite (cmd/status.go:322-327).
CleanupRelations stats both endpoints of every relation, so one document delete costs up to 2R stats (internal/sync/manifest.go:297-312).
Hooks call LoadManifest directly without the MCP store cache. A cold load validates every relation and formats a string per relation (internal/sync/manifest.go:225).
Only read-tool benchmarks exist (internal/mcp/tools/scaling_bench_test.go, realistic_bench_test.go).
Preliminary numbers
These come from a throwaway benchmark on a copy of manifest.go (Apple M5, APFS, no fsync, about 204 bytes per relation):
R
clone + add + save
file
AddRelation scan
cold LoadManifest
500
0.45 ms
102 KB
1.4 µs
1.1 ms
5,000
4.1 ms
1.0 MB
11 µs
11 ms
50,000 (maxManifestRelations)
30.5 ms
10.2 MB
112 µs
105 ms, 70 MB, 350k allocs
At R = 50,000, bulk-adding 1,000 edges costs about 30 s of rewriting. At this repository's size (326 relations, 58 KB), each write is under 1 ms.
The linear duplicate scan in AddRelation is negligible next to the save. A key-set dedup would not change the result, so it is dropped as a candidate.
Scope
Add in-repo benchmarks in the style of internal/mcp/tools/*_bench_test.go for: add_relation end to end, SaveManifest, cold LoadManifest, and remove_document with CleanupRelations, at R = 500, 5,000, 50,000.
Record the results in .archcore/mcp/relation-write-path-graph-growth.idea.md.
Decide from the numbers whether any of these is worth doing: a batch add_relations tool (weigh the extra tool-definition tokens, see .archcore/cli/mcp-token-optimization.idea.md), a cheaper CleanupRelations, a cached or lighter validation in LoadManifest for hooks.
Acceptance: benchmarks are committed and the decision is recorded. Optimizations, if any, get their own issues.
Summary
Split out of #26. The relation write path rewrites the whole
.archcore/.sync-state.jsonon every mutation. Before choosing an optimization, add benchmarks so the decision rests on measurements.Current behavior (v0.8.3)
manifestStore.mutateclones the manifest and callsSaveManifest(internal/mcp/tools/manifest_store.go:87,91).SaveManifestre-marshals the whole file withMarshalIndent(internal/sync/manifest.go:327).add_relation,remove_relation, andremove_documentall go throughmutate.archcore statusdoes the same full rewrite (cmd/status.go:322-327).CleanupRelationsstats both endpoints of every relation, so one document delete costs up to 2R stats (internal/sync/manifest.go:297-312).LoadManifestdirectly without the MCP store cache. A cold load validates every relation and formats a string per relation (internal/sync/manifest.go:225).internal/mcp/tools/scaling_bench_test.go,realistic_bench_test.go).Preliminary numbers
These come from a throwaway benchmark on a copy of
manifest.go(Apple M5, APFS, no fsync, about 204 bytes per relation):AddRelationscanLoadManifestmaxManifestRelations)At R = 50,000, bulk-adding 1,000 edges costs about 30 s of rewriting. At this repository's size (326 relations, 58 KB), each write is under 1 ms.
The linear duplicate scan in
AddRelationis negligible next to the save. A key-set dedup would not change the result, so it is dropped as a candidate.Scope
internal/mcp/tools/*_bench_test.gofor:add_relationend to end,SaveManifest, coldLoadManifest, andremove_documentwithCleanupRelations, at R = 500, 5,000, 50,000..archcore/mcp/relation-write-path-graph-growth.idea.md.add_relationstool (weigh the extra tool-definition tokens, see.archcore/cli/mcp-token-optimization.idea.md), a cheaperCleanupRelations, a cached or lighter validation inLoadManifestfor hooks.Acceptance: benchmarks are committed and the decision is recorded. Optimizations, if any, get their own issues.