Skip to content

Fix/forest type document statistic - #642

Open
tsubik wants to merge 5 commits into
hot-fix/documents-being-soft-deletedfrom
fix/forest-type-document-statistic
Open

Fix/forest type document statistic#642
tsubik wants to merge 5 commits into
hot-fix/documents-being-soft-deletedfrom
fix/forest-type-document-statistic

Conversation

@tsubik

@tsubik tsubik commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Speed up document statistics counting

generate_for_country_and_day counted in Ruby: the join-alias columns
(fmus.forest_type, required_operator_documents.required_operator_document_group_id)
went through method_missing at ~33µs per read vs ~0.31µs for a real attribute,
the triple loop rescanned every document once per combination, and
previous_stat ran one query per combination.

Now Postgres does the counting — a single GROUPING SETS query per day covers
every country and every rollup — and previous stats load in one DISTINCT ON.

Measured over a full rebuild:

  • unoptimized, with the forest-type fix applied: ~208s per day across all six
    scopes (~60h for 1,049 days)
  • after this PR: 0.43s per day — 497 days rebuilt in 3.6 min

Worth noting the two are linked: the old code was only fast because the broken
forest-type comparison short-circuited on the first check. Fixing it makes far
more comparisons succeed, which made the unoptimized version dramatically
slower — so this optimization is what keeps the fixed code usable.

Output is unchanged. Rebuilt 2023-03-08 → today under the old and new
implementations through an identical loop and compared with EXCEPT ALL both
ways: 42,538 rows each, zero differences — including the set of rows whose date
was moved forward.

Also included: forest-type slices in statistics were always zero (same
String/Integer mismatch, broke when Fmu.forest_type became an enum in March
2023), sync:operator_documents_stats swept with delete_old: false so slices
stopped summing to their rollups, and a one-off fix:erroneous_document_deletion
task for the two documents this bug deleted.

🤖 Generated with Claude Code

@tsubik
tsubik requested a review from santostiago September 2, 2026 16:49
@tsubik
tsubik force-pushed the fix/forest-type-document-statistic branch from 83a1daf to 64482af Compare September 2, 2026 16:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant