CRDC CDE alignment, DataCatalog/SCDM integration, and kg-pipeline buildout

.gitattributes

@@ -0,0 +1,3 @@
+* text=auto eol=lf
+
+*.csv text eol=lf

.gitignore

@@ -78,4 +78,5 @@ kg-pipeline/data/mc2_assay/rdf/*
 kg-pipeline/__pycache__/
 kg-pipeline/**/__pycache__/
 kg-pipeline/.venv/
-.pytest_cache/
+.pytest_cache/
+/.vscode

CLAUDE.md

@@ -0,0 +1,87 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## What this repo is
+
+Data models and controlled vocabularies for the [Cancer Complexity Knowledge Portal](https://cancercomplexity.synapse.org/) (CCKP). The model is maintained as CSV files in domain-specific `modules/`, collated into `mc2.model.csv`, and converted to JSON-LD/JSON Schema via `synapseclient.extensions.curator` (the actively-maintained successor to [schematicpy](https://pypi.org/project/schematicpy/), which Sage Bionetworks has announced will be retired by end of 2026) for use by the [Data Curator App](https://dca.app.sagebionetworks.org/).
+
+A second subsystem, `kg-pipeline/`, converts this model (plus live CCKP portal data pulled from Synapse) into an RDF knowledge graph - its own LinkML schema, an extract → harmonize → map-to-RDF → validate build, and a Synapse publish path. It has its own README, Makefile, and Python environment - see "kg-pipeline (knowledge graph)" under Architecture below.
+
+## Commands
+
+```bash
+# Install dependencies (Python 3.10+)
+pip install -r requirements.txt
+
+# Full build: update valid values in all modules → collate → generate JSON Schemas
+make all
+
+# Steps individually:
+python update_valid_values.py   # reads modules/mapping.yaml, rewrites annotationProperty.csv Valid Values columns
+make collate                    # concatenates all modules/*/annotationProperty.csv → mc2.model.csv
+make convert                    # python convert_model_to_jsonld.py → mc2.model.jsonld
+make generate-json              # python create_json_from_model.py <data types> → json_schemas/
+
+# Generate JSON schemas for specific data types only
+python create_json_from_model.py Biospecimen Study Dataset
+
+# Docs dev server
+mkdocs serve   # http://localhost:8000
+```
+
+```bash
+# kg-pipeline: separate venv + requirements.txt (isolated from the root
+# install above - linkml pulls a large, independently-versioned dependency
+# tree). Run from kg-pipeline/.
+cd kg-pipeline
+python3 -m venv .venv && source .venv/bin/activate
+pip install -r requirements.txt
+
+make schema     # regenerate schema/*.ttl from schema/*.linkml.yaml
+make all        # schema + extract (needs Synapse credentials) + harmonize + triples + validate
+make full-kg    # combined-kg + SCDM/ontology-crosswalk federation -> data/rdf/cckp_kg_full.ttl
+make test       # pytest test/ - no live Synapse access needed
+```
+
+## Architecture
+
+### The module + collation pattern
+
+Each domain lives in `modules/<domain>/`:
+- `annotationProperty.csv` — attribute definitions for that domain (type, description, valid values, validation rules)
+- One CSV per controlled vocabulary (e.g., `specimenType.csv`, `fixative.csv`) — the actual enumerated terms
+
+`modules/mapping.yaml` is the central registry: it maps each attribute name to the CV CSV that provides its valid values. `update_valid_values.py` reads this file and rewrites the `Valid Values` column in each `annotationProperty.csv`.
+
+`make collate` then concatenates all `annotationProperty.csv` files into `mc2.model.csv` (the header comes from the consortium module, body from all others via `tail -n +2`).
+
+### CSV conventions
+- All columns read/written as `dtype=str` — `TRUE`/`FALSE` must not become `True`/`False`
+- No NaN — use empty strings (`keep_default_na=False`)
+- Index on `Attribute` column when updating via pandas
+
+### kg-pipeline (knowledge graph)
+
+`kg-pipeline/` is a separate, self-contained pipeline (own `README.md`, `Makefile`, `requirements.txt`, `test/`) that builds an RDF knowledge graph from this model plus live CCKP portal data:
+
+- **Stage 0**: this repo's `mc2.model.csv` → `schema/mc2_model.linkml.yaml` (via a vendored `csv-to-linkml` converter) → `schema/mc2_model.ttl` (OWL/Turtle)
+- **Stages 2-4**: pull the 5 CCKP portal Synapse tables → resolve controlled-vocabulary values against this model's CVs (real ontology IRIs: NCIT, MONDO, EFO, OBI, ...) → mint RDF instance triples typed `cckp:{Class}`, at a placeholder `w3id.org/mc2-center/cckp-portal/data/{Class}/{id}` IRI
+- **Stage 5**: SHACL shape validation + `queries/*.rq` sanity checks
+- Optional stages: Data Catalog annotations, SCDM (Sage Common Data Model) federation, MC2 assay-metadata KG (access-controlled, linked into `sagebrain-model`)
+- Publish path: Synapse only (`make publish-portal-kg`, `make deploy-kg`)
+
+See `kg-pipeline/README.md` for the full command reference, directory layout, and design rationale (also `plans/kg_pipeline_architecture_decisions.md`).
+
+### PR requirements
+PRs to main must have exactly one semantic label: `major`, `minor`, `patch`, or `non-release`. The `pr-check.yml` workflow enforces this.
+
+### CI workflows (`.github/workflows/`)
+| Workflow | Trigger | What it does |
+|----------|---------|--------------|
+| `build-docs.yml` | Push to main | Builds MkDocs site → GitHub Pages |
+| `pr-check.yml` | PR events | Validates semantic label |
+| `google-sheet-sync.yml` | Scheduled/manual | Syncs RFC Google Sheets to a CSV branch |
+| `create-release.yml` | Manual trigger | Creates GitHub release with version bump |
+
+No CI currently validates `make all` (root model) or `kg-pipeline/`'s build/tests on PRs (the old `build-jsonld.yml` was removed as obsolete; nothing replaced it) - run them locally before pushing model or kg-pipeline changes.

Makefile

@@ -1,10 +1,7 @@
 CSV := mc2.model.csv
-QC := ./qc_model/mc2_qc.model.csv
 DATA := DataDSP Study FileView PublicationView GrantView ToolView EducationalResource DatasetView DataCatalog Biospecimen Model Individual SequencingLevel1 SequencingLevel2 SequencingLevel3 SequencingRNALevel1 ImagingLevel1 ImagingLevel2 ImagingLevel3Image ImagingLevel3Segments ImagingLevel4 NanoStringGeoMxAuxiliaryFiles NanoStringGeoMxDSPImaging NanoStringGeoMxDSPLevel1 NanoStringGeoMxDSPLevel2 NanoStringGeoMxDSPLevel3 NanoStringGeoMXROISegmentAnnotation 10xVisiumAuxiliaryFiles 10xVisiumRNALevel1 10xVisiumRNALevel2 10xVisiumRNALevel3 10xVisiumRNALevel4
 
-all: collate generate-json
-
-qc: collate qc_convert
+all: collate convert generate-json
 
 collate:
 	@echo "Collating module components..."
@@ -13,10 +10,7 @@ collate:
 	tail -n +2 -q modules/*/annotationProperty.csv >> ${CSV}
 
 convert:
-	schematic schema convert ${CSV}
-
-qc_convert:
-	schematic schema convert ${QC}
+	python convert_model_to_jsonld.py
 
 generate-json:
 	python create_json_from_model.py ${DATA}

README.md

@@ -26,41 +26,90 @@
 
 🔎 **Data Models Explorer**: https://mc2-center.github.io/data-models/
 
-📊 **Data Curator App**: https://dca.app.sagebionetworks.org/
-
 ---
 
 ## Overview
 
 This project contains the released versions of the JSON-LD schemas for the
 [Cancer Complexity Knowledge Portal] (CCKP), and more broadly, MC2 Center.
 You can learn more about the schemas/data models and other aspects of this
-project in our portal documentation - coming soon! The MC2 Center data model
+project in our Data Models Explorer. The MC2 Center data model
 is in both CSV and JSON-LD format, and individual entity schemas are also
 exported as standalone JSON Schemas in `./json_schemas`.
 
 Beyond the original entity types (Dataset, Study, Publication, Grant,
 Educational Resource, File, Tool, Person), the model also covers Biospecimen,
-Individual, and Model (non-human organism/cell line) entities, as well as
-assay-level metadata for imaging (multiplexed/single-channel imaging),
-NanoString GeoMx Digital Spatial Profiler (DSP) spatial transcriptomics,
-bulk/single-cell sequencing, and 10x Genomics Visium spatial transcriptomics.
+Individual, Model (non-human organism/cell line), and Data Catalog entities,
+as well as assay-level metadata for imaging (multiplexed/single-channel
+imaging), NanoString GeoMx Digital Spatial Profiler (DSP) spatial
+transcriptomics, bulk/single-cell sequencing, and 10x Genomics Visium
+spatial transcriptomics.
+
+A companion [`kg-pipeline`](./kg-pipeline) project builds an RDF knowledge
+graph from this model and live CCKP portal data - see
+[Knowledge Graph Pipeline](#knowledge-graph-pipeline) below.
+
+## Usage
+
+Requires Python 3.10+.
+
+```bash
+pip install -r requirements.txt
+
+# Full build: update valid values in all modules -> collate -> generate JSON Schemas
+make all
+```
+
+Steps individually:
+
+```bash
+python update_valid_values.py   # reads modules/mapping.yaml, rewrites annotationProperty.csv Valid Values columns
+make collate                    # concatenates all modules/*/annotationProperty.csv -> mc2.model.csv
+make convert                    # converts mc2.model.csv -> mc2.model.jsonld
+make generate-json              # python create_json_from_model.py <data types> -> json_schemas/
+
+# Generate JSON schemas for specific data types only
+python create_json_from_model.py Biospecimen Study Dataset
+```
+
+To build and preview the docs site locally:
+
+```bash
+mkdocs serve   # http://localhost:8000
+```
+
+See [contributing guidelines] for the full development and release process.
+
+## Knowledge Graph Pipeline
+
+[`kg-pipeline/`](./kg-pipeline) is a separate, self-contained pipeline that
+converts this model plus live CCKP portal data (pulled from Synapse) into a
+queryable RDF knowledge graph: an [extract → harmonize → map-to-RDF →
+validate](./kg-pipeline#architecture) build producing a LinkML/OWL schema
+and per-entity RDF instance data, with real ontology IRI mappings (NCIT,
+MONDO, EFO, OBI, ...) sourced from this model's own controlled vocabularies.
+It also links the graph to related efforts - Data Catalog (native Synapse
+Dataset annotations), SCDM (Sage Common Data Model) federation, and
+[sagebrain-model](https://github.com/Sage-Bionetworks/sagebrain-model)
+interoperability for assay-level metadata.
+
+It has its own README, `Makefile`, and Python environment (isolated from
+this repo's root `requirements.txt`). See
+**[`kg-pipeline/README.md`](./kg-pipeline/README.md)** for setup, the full
+command reference, and design rationale.
 
 ## Folder Structure
 
 ```
 .
-├── dca_config/
 ├── docs/
+├── json_schemas/
+├── kg-pipeline/
 ├── modules/
 ├── scripts/
 └── templates/
 ```
 
-### DCA Configuration
-
-MC2 Center's configurations for the DCA is located in `./dca_config`.
-
 ### Documentation
 
 All docs are located in the `./docs` directory and are written in Markdown