SorghumBase search-index build (build/)

A robust, re-runnable orchestration of the Gramene/SorghumBase gene search-index ETL across the five repos in this directory. It replaces the hand-followed recipe with idempotent, validated, logged stages that are safe to re-run and easy to update when new data arrives (new genomes, new compara, new expression, …).

Target of this build (read from gramene-mongodb-config/collections.js, the single source of truth): mongo db sorghum11, solr cores sorghum_genes11 and sorghum_suggestions11. Version 11 was chosen so the build never touches the live release-10 data (sorghum10, sorghum_genes10, sorghum_suggestions10).

Quick start

cd build
make all          # full build: mongo collections + solr genes + suggestions
make status       # see which stages are done

To include EBI Expression Atlas data in the index:

make 55_atlas          # downloads + loads experiments/assays/expression
make refresh-expression  # (or this — also re-runs solr genes to pick it up)

Run a single stage: make 25_reactome (or bash stages/25_reactome.sh). Every stage tees to logs/<stage>.log and, on success, stamps .state/<stage>.done.

Design principles (why this is more robust than the recipe)

1. One source of truth for the target db. config.sh reads getMongoConfig() from collections.js, so the db name/version can never drift from what the ETL code writes. All consumer repos now depend on the local config via file:../gramene-mongodb-config instead of a mutable GitHub branch — the old #sorghum_v10_fable / #sorghum_v7 pins resolved to oryza7 / sorghum7 and would have silently built into the wrong db.
2. Every stage is idempotent. Collections that the underlying scripts only insert into (qtls, domains, genetree, genes, experiments/assays/expression) are dropped first; drop-based loaders (maps, GO/PO/TO, pathways) are re-run as is. Solr cores are recreated fresh (no stale/orphaned docs). Re-running any stage reproduces the same result.
3. Every stage validates and fails loudly. Counts are checked against expectations; the gene dumps are gated against maps.num_genes before ingestion (the recipe's "moral of the story" check, now enforced); empty JSON / empty facet outputs abort instead of silently shipping a half-built index.
4. Resumable + targeted refresh. Stamps let make skip completed stages. make refresh-compara|refresh-genes|refresh-expression|refresh-reactome|refresh-ontologies invalidate exactly the stages affected by a given upstream change and rebuild.

Stage graph

05_install ─ 00_preflight ─ 10_maps ┬ 15_ontologies ┐
                                    ├ 20_variation  │
                                    ├ 25_reactome   ├─ 50_genes_decorate ─ 52_tree_domains
                                    ├ 30_curated ─ 35_genetrees             │
                                    └ 40_genes_dump                        │
                          00_preflight ─ 45_homologs ──────────────────────┤
                                    (55_atlas, optional) ──────────────────┤
                                                                 50 ─ 60_solr_genes ─ 65_solr_suggestions ─ 70_services

stage	builds	source	idempotency
05_install	node_modules (local config)	npm	safe
00_preflight	(validation only)	—	safe
10_maps	maps, taxonomy	mysql cores/compara	drop-based
15_ontologies	GO, PO, TO, domains	OBO + InterPro	drop-based
20_variation	qtls	mysql variation	drops first
25_reactome	pathways, taxonomy.reactomePrefix, genes_to_pathways.json	Plant Reactome	drop-based
30_curated	curated/curated.json	thalemine/rapdb + redis db3	overwrite
35_genetrees	genetree	mysql compara + curated.json	drops first
40_genes_dump	search/tmp/*.json.gz	mysql cores	resumable; gated vs maps
45_homologs	homologs	mysql compara	drop-based
50_genes_decorate	genes (+ indexes)	tmp dumps + all aux collections	drops first
52_tree_domains	genetree (domains on leaves)	genes + domains	updateOne
55_atlas (optional)	experiments, assays, expression	EBI Atlas	drops first
60_solr_genes	sorghum_genes11	mongo genes (+VEP opt)	recreates core
65_solr_suggestions	sorghum_suggestions11	genes core + mongo	recreates core
70_services	swagger/ebeye config	—	idempotent edits

Prerequisites (verified by 00_preflight)

• mongo, solr (http://localhost:8983), redis, ensembl REST (https://data.gramene.org/pansite-ensembl-115 — the public pansite service)
• mysql access to host cabot (compara + 128 cores + variation)
• ensembl_db_info.json with 128 cores, 9 anchor flags, 2 variations, the right compara
• GeneRIFs in redis db 3 — loaded separately from NCBI (see below); reused across builds.

Source fixes applied (genuine bugs the recipe glossed over)

• gramene-mongodb, gramene-solr, gramene-swagger package.json → config via file:../gramene-mongodb-config (was stale GitHub branches → wrong db).
• search/decorate.js required ./addQtlXrefs (missing) → now ./qtl_adder.
• search/msu6_adder.js hard-required a missing rice LUT → now optional passthrough.
• createDumpCommandsfromCores.js, ontologies/populate.js, curated/get_curated.js, reactome/get_species_prefixes.js, reactome/get_pathways.js, search/dump_homologs.js left the config's mongo connection open and hung after finishing → each now closes/exits cleanly.
• search/curated.js fetch (in the decorate hot path) had no timeout → added a 60s abort + empty-LUT fallback so a network stall can't hang the whole decorate.
• Removed the unused xml-stream dep (its native node-expat broke npm install).

Incremental updates (new Atlas studies)

Stages are full rebuilds by default. Where an update genuinely affects only part of the index, there is an explicit incremental entry point — the stages never guess which mode is wanted, since a wrong guess would ship a partially-updated index with a zero exit code. Each incremental mode validates its preconditions and dies naming the full target if they do not hold.

make add-studies ACCESSIONS="E-MTAB-8969,E-MTAB-10280"

55_atlas (ATLAS_ADD=) upserts only those experiments; 58_expression_attributes (EXPR_ONLY=) rebuilds only the affected genomes; 62_attr_atomic expression patches the genes core in place. This works because the expression collection is keyed on gene id with one field per experiment, so adding a study is a $set — the loaders (parseData.js, project_expression_via_lut.js, getAtlasData.js) are upsert-based and idempotent, which also means a re-run of the full 55_atlas is no longer destructive.

Granularity is per genome, not per gene: a new study shifts that genome's reference quantiles (ref_<taxon>.json), which changes scores for every gene of that genome.

Gene attribute capabilities (MAKER / VEP / Grassius)

60_solr_genes merges gene-level attribute tables into the solr docs via add_attributes.pl (chained), then loads. Re-do just this layer without re-decorating: make refresh-attributes.

• MAKER — per-genome gene-model AED/QI quality scores (MAKER capability, MAKER__*__attr_[fi]). Table: ${MAKER_TABLE} (defaults to the prior release's maker_attrib_table_for_solr.txt, built from /scratch/olson/28_sorg_maker_AED/ *.maker.gene_only.AED_QI.scores via munge_MAKER.pl). Same gene IDs across releases, so reused as-is; regenerate + repoint MAKER_TABLE for new annotations.
• VEP / germplasm — loss-of-function / PTV alleles per population (VEP capability, VEP__<consequence>__<homo|het>__<pop>__attr_ss + VEP__merged__{EMS,NAT}__attr_ss). Table: ${VEP_TABLE} (prior release's vep_attributes.txt). Built by vep/dump_gene_level_VEP_table.pl --registry <reg.pm> -s <species> → vep/munge_vep.pl. The sorghum variation DB (sorghum_bicolor_variation_9_108_30) is unchanged, so reused as-is; regenerate for a new variation DB.
• Grassius (direct) — TF family on the reference genomes, applied in decorate from search/grassius.tsv (Grassius__xrefs). Pull a fresh copy from grassius.org (old assembly IDs mapped to current gene IDs via synonyms, merged so curated coverage is never lost) with make refresh-grassius (regenerates grassius.tsv, then re-decorates + rebuilds solr). NOTE: the grassius.org download currently has less sorghum coverage than the curated file, which is why we merge rather than replace.
• grassius_homolog — Grassius TF families projected onto homologs via gene trees (grassius_homolog__attr_ss), so the 124 pan-genome assemblies (whose gene IDs aren't in Grassius) still get a TF-family annotation. Generated fresh each build by build/grassius_homolog_table.js and merged in 60_solr_genes.

Things still done out-of-band

• GeneRIFs (redis db 3 / db 2). Download generifs_basic from NCBI and load per the scripts in ~olson/src/.../genetrees/database/update_geneRIFs.README.txt. db2 = entrezgene-keyed, db3 = ensembl-id-keyed (what decorate reads). The build reuses whatever is in db3; refresh it before a release if you want current RIFs.
• Reactome japonica enhancement (optional). The recipe appended UniProt→gene mappings for Oryza sativa japonica to Ensembl2PlantReactomeReactions.txt using merged_lut.txt. Skipped here (rice-japonica-specific). To apply, do the awk/pack munge from the recipe before 25_reactome's get_pathways.js.
• Regenerating VEP/MAKER tables (only when the variation DB or MAKER run changes): VEP via vep/dump_gene_level_VEP_table.pl + munge_vep.pl (needs the Ensembl Perl API + a registry); MAKER via /scratch/olson/28_sorg_maker_AED/munge_MAKER.pl. Then point MAKER_TABLE/VEP_TABLE at the new files. Otherwise both are reused as-is.
• Deployment (apache proxy, firewall, ensembl REST registry, BLAST dumps): see the checklist printed by 70_services — infra steps the build won't auto-apply.

Build invariants enforced by the stages

Each of these exists because a violation shipped, or nearly shipped, a silently wrong index. They fail the build rather than warn.

• 05_install relinks gramene-mongodb-config. npm copies the file:../ dependency instead of symlinking it, so edits to collections.js are invisible to the ETL and a newly added collection comes back undefined deep inside a later stage. The stage forces a symlink after npm runs; 00_preflight additionally asserts every required collection resolves.
• 40_genes_dump gates dumps against maps. Gene docs embed taxon_id, and the synthetic per-genome taxon ids are reassigned whenever the genome set changes — with gene counts unchanged, so the line-count check cannot see it. Dumps older than .state/10_maps.done are re-dumped, and every dump's taxon_id must equal maps.taxon_id for that genome.
• 65_solr_suggestions gates suggestions against the genes core. Both suggestion layers are derived from that core; a well-formed but stale genes.json once produced ~170k suggestions for genes that no longer existed. Inputs older than .state/60_solr_genes.done are regenerated, and the primary-id count must equal the genes core.
• decorate reports stalls instead of hanging. Most adders are promise.then(... done()) with no rejection handler, so a throw leaves done() uncalled and deadlocks the pipeline with no output. Per-stage counters plus a watchdog name the stage that swallowed a gene and exit; ancestor_adder and homolog_adder surface their own errors.

Gene-tree stable ids

Trees whose compara stable_id is NULL get a synthetic id SB<version>GT_<root_id>, derived from collections.getVersion() (so v11 emits SB11GT_). It was previously hardcoded and labelled v11 trees with the release-10 prefix. Override with GT_PREFIX if a release needs another scheme.

Tunables (env vars)

DUMP_PARALLELISM (default 8) · SOLR_URL · ENSEMBL_REST · MAKER_TABLE · VEP_TABLE · REBUILD_CORE=1 (force-recreate the genes core) · GRASSIUS_SPECIES (override 28_grassius_source species list) · SWAGGER_PORT/EBEYE_PORT/START_SERVICES (70_services).