Fix/forest type document statistic - #642
Open
tsubik wants to merge 5 commits into
Open
Conversation
tsubik
force-pushed
the
fix/forest-type-document-statistic
branch
from
September 2, 2026 16:54
83a1daf to
64482af
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Speed up document statistics counting
generate_for_country_and_daycounted in Ruby: the join-alias columns(
fmus.forest_type,required_operator_documents.required_operator_document_group_id)went through
method_missingat ~33µs per read vs ~0.31µs for a real attribute,the triple loop rescanned every document once per combination, and
previous_statran one query per combination.Now Postgres does the counting — a single
GROUPING SETSquery per day coversevery country and every rollup — and previous stats load in one
DISTINCT ON.Measured over a full rebuild:
scopes (~60h for 1,049 days)
Worth noting the two are linked: the old code was only fast because the broken
forest-type comparison short-circuited on the first check. Fixing it makes far
more comparisons succeed, which made the unoptimized version dramatically
slower — so this optimization is what keeps the fixed code usable.
Output is unchanged. Rebuilt 2023-03-08 → today under the old and new
implementations through an identical loop and compared with
EXCEPT ALLbothways: 42,538 rows each, zero differences — including the set of rows whose date
was moved forward.
Also included: forest-type slices in statistics were always zero (same
String/Integer mismatch, broke when
Fmu.forest_typebecame an enum in March2023),
sync:operator_documents_statsswept withdelete_old: falseso slicesstopped summing to their rollups, and a one-off
fix:erroneous_document_deletiontask for the two documents this bug deleted.
🤖 Generated with Claude Code