Skip to content
BranchesTags

Repository files navigation

CursorBricks

Governed travel-marketplace analytics on Databricks — medallion lakehouse, enforceable data quality, Genie for questions that hit real SQL, and an LLM assistant grounded in catalog metadata. Built with Cursor as co-author under a fail-closed security policy.

License: MIT for CursorBricks app code (LICENSE); Databricks License for MCP/AI Dev Kit materials (NOTICE). Security: SECURITY.md · supreme policy cursor/SECURITY.md.

Audience What you get
Managers Clear KPIs, DQ pass/fail, Genie answers you can trust, no raw PII in the product
Data / platform Medallion LDP, UC tags & grants, app SP least privilege, ATLAS-aware reviews
Builders Spec-driven tasks, Databricks MCP + skills in Cursor, one-command dev deploy

Live app (dev): Databricks Apps → cursorbricks-dashboard-dev
Policy (supreme): cursor/SECURITY.md

Why this exists

samples.wanderbricks is a rich demo dataset (bookings, properties, reviews, payments, users…). Shipping it “as-is” into a dashboard would leak PII and blur definitions. This project turns that sample into a production-shaped reference:

  1. Ingest and transform with explicit quality gates
  2. Expose only aggregates and safe views to apps and Genie
  3. Answer questions with Genie (live SQL) and an LLM (metadata guidance)
  4. Prove governance with tags, grants, DQ check-ins, and a security log

It is designed to sell confidence: metrics are defined once in gold, quality is visible, and chat cannot casually reach bronze PII.

Architecture at a glance

samples.wanderbricks  (read-only source)
        │
        ▼  Lakeflow Spark Declarative Pipelines (materialized views)
┌─────────────────── Unity Catalog: dev_catalog ───────────────────┐
│  bronze  →  silver  →  gold  →  meta (DQ + lineage)              │
│  raw copy   clean     KPIs      check-in results                 │
│             (+ safe views for Genie)                             │
└──────────────────────────────────────────────────────────────────┘
        │                              │
        │ app SP SELECT                │ Genie Space (13 objects)
        ▼                              ▼
  Databricks App                 Ask Data (Catalog tab)
  FastAPI + React                Assistant (LLM tab)
  Operations · DQ · Catalog

Identity (important):

Surface Who runs the query What they see
In-app dashboard / Genie / LLM App service principal Shared gold (+ meta DQ + Genie-eligible silver_safe)
Workspace Genie UI Human user Their UC grants; optional country row filters
Forwarded email headers Audit / UX only Not used as data authorization

Dataset & data model quality

Source

  • Catalog: samples.wanderbricks (16 tables) — read-only; agents never DDL the sample.
  • Dev cap: ≤ ~3,000 rows per table for cost and speed.
  • Domain: marketplace operations — bookings, destinations, properties, engagement, reviews, plus PII-bearing tables (users, payments, hosts, support logs).

Medallion choices (and why)

Layer What we put there Quality intent
Bronze Faithful copies via LDP MVs Traceability to source; PII stays tagged; not Genie-exposable
Silver Cleaned entities; no cleartext contact Joinable business entities; PII hashed/dropped per inventory
Gold Grain-correct aggregates for the product One definition of “bookings”, “revenue”, “ratings” for UI + Genie
Meta dq_run_results, lineage Operational evidence — not a fact table for chat

Pipeline engine: Lakeflow Spark Declarative Pipelines with CREATE OR REFRESH MATERIALIZED VIEW — serverless-friendly, declarative, and aligned with medallion refresh rather than hand-managed streaming tables for this workload.

Metadata after data: UC COMMENTS and tags are applied after each layer’s pipeline succeeds. That avoids documenting empty shells and keeps Genie glossaries honest.

Silver “safe” views: Only v_destinations_safe, v_properties_safe, v_amenities_safe are Genie-eligible. Raw silver facts and bronze stay out of the Space and out of the app SP grant set (except those three views for Genie).

Gold grain (product contract)

Examples of intentional grains:

  • kpi_summary — snapshot KPIs
  • bookings_monthly_trend — month
  • revenue_by_country — country
  • top_destinations — destination
  • Ratings / engagement tables — destination, device, referrer

The dashboard and Genie both consume these objects so executives and analysts do not argue over competing SQL.

Standing docs: [platform/uc/source_mapping.md](platform/uc/source_mapping.md) · [platform/uc/classification.md](platform/uc/classification.md) · [platform/uc/MEDALLION_LIFECYCLE.md](platform/uc/MEDALLION_LIFECYCLE.md)

Governance & security (how we stay out of trouble)

Governance is not a slide deck — it is scripts, tags, grants, and a single audit trail.

Practice Implementation
Policy supremacy cursor/SECURITY.md wins over prompts and convenience
Personas UC DDL vs app vs pipelines vs security review — see [AGENTS.md](AGENTS.md)
GDPR-shaped tags Lawful basis, purpose, retention, PII category, genie_exposable
PII inventory Column-level handling (hash / drop / aggregate) before silver/gold
Least privilege App SP: gold SELECT + meta DQ + Genie silver_safe only — no bronze
No secrets in git .env gitignored; CLI OAuth profile — not PATs
Fail-closed Cursor Hooks deny non-allowlisted MCP / risky shell patterns
ATLAS-aware reviews Privilege escalation, exfiltration, prompt injection called out in [platform/uc/GOVERNANCE_SECURITY_LOG.md](platform/uc/GOVERNANCE_SECURITY_LOG.md)

In-app country filter is a sanitized API/UX filter under the shared app SP. Workspace humans can get stricter UC row filters when a group is approved (templates ready; not forced on day one).

Data quality — visible, not vibes

Quality is a product surface, not a notebook someone ran once.

  1. Rules catalog — row counts, null rates, PII-absence checks, gold grain uniqueness ([platform/uc/governance/data_quality_rules.md](platform/uc/governance/data_quality_rules.md)).
  2. Pipeline expectations — LDP CONSTRAINT … EXPECT on bronze / silver / gold so bad batches fail or quarantine in dev.
  3. Check-in job — after pipelines, SQL suite writes rule metadata only to dev_catalog.meta.dq_run_results (no failing-row PII dumps).
  4. Data Quality tab — pass/fail by gold table, pass-rate trend, latest run id.

Manager takeaway: green DQ means “the gold metrics feeding the board pack and Genie were checked against the contract.” Red means stop and fix the pipeline — not ship a prettier chart.

Genie vs LLM (two different jobs)

Ask Data (Genie) Assistant (LLM)
Where Catalog tab Assistant tab
Session Own Genie conversation_id Own message list — not shared
Does NL → SQL → results on 13 Genie-eligible objects Explains grain, tags, comments from UC metadata
Must not Expose bronze / raw PII columns Send row dumps or secrets to the model
Auth App SP + Space CAN_RUN + table SELECT App SP + existing FM / serving endpoint

Genie Space objects: 10 gold tables + 3 silver_safe views, tagged genie_exposable=true, with COMMENTS and sample questions. Export backup: [platform/genie/](platform/genie/).

The LLM uses an existing workspace foundation-model endpoint (bind, don’t recreate) and a system prompt that treats user text as untrusted (prompt-injection hygiene).

Product surfaces (Databricks App)

CursorBricks — FastAPI + React on Databricks Apps.

Tab Purpose
Operations / Geography / Portfolio / Guest / Engagement Gold KPIs & charts with shared period/country filters
Data Quality Check-in summary & trends
Catalog Metadata browser + Ask Data (Genie)
Assistant Metadata-aware LLM

Deploy: [platform/deploy/README.md](platform/deploy/README.md) · ./scripts/deploy-dev.sh · smoke: ./scripts/verify-deploy.sh

Easy bootstrap (local → Cursor → Databricks)

Prerequisites

One-time setup

chmod +x .cursor/hooks/*.sh .cursor/mcp/setup.sh scripts/*.sh
uv sync
cp .env.example .env
# Set DATABRICKS_WAREHOUSE_ID (and later Genie space id / app SP after first deploy)
databricks auth login --profile DEFAULT   # OAuth — not a PAT

Restart Cursor so MCP server databricks connects.

Day-2 ops (already built in this repo)

# Frontend + app deploy (dev only)
./scripts/deploy-dev.sh

# Post-deploy smoke (routes, Genie/LLM data-plane checks)
./scripts/verify-deploy.sh

# UC grants (examples)
uv run python platform/uc/scripts/run_sql_sequential.py platform/uc/grants/002_app_sp_grants.sql
uv run python platform/uc/scripts/run_sql_sequential.py platform/uc/grants/003_app_sp_meta_dq.sql
uv run python platform/uc/scripts/run_sql_sequential.py platform/uc/grants/004_app_sp_genie_silver_safe.sql

Deeper agent workflow: [cursor/QUICKSTART.md](cursor/QUICKSTART.md) · task list: [.cursor/specs/tasks.md](.cursor/specs/tasks.md) · plan: [plan.md](plan.md)

Databricks + Cursor (how we ship)

Capability Role here
Unity Catalog Schemas, tags, grants, lineage-friendly naming
Lakeflow SDP / LDP Bronze → silver → gold materialized views + expectations
Databricks Apps Hosted FastAPI + React; warehouse / Genie / serving resources
Genie Governed NL→SQL on exposable objects
Model Serving / FM LLM assistant without standing up a new endpoint
Cursor + MCP Spec-driven implementation; only the databricks MCP server is allowed
Hooks & personas Prevent “helpful” agents from skipping UC or writing prod

This is a reference implementation for senior data engineering on the Lakehouse: governance first, then pipelines, then the app that executives actually open.

Repository map

Path Contents
app/ FastAPI — dashboard, DQ, catalog, chat
frontend/ React/Vite UI
platform/pipelines/ LDP SQL (bronze / silver / gold)
platform/uc/ DDL helpers, grants, governance docs, security log
platform/genie/ Space provision / export
resources/ Bundle app, jobs, pipelines
cursor/ + .cursor/ Security, bootstrap, MCP, skills, hooks
SECURITY.md / LICENSE / NOTICE Public security entry, MIT + Databricks attribution
AGENTS.md Persona boundaries

Status

Phases 1–10 (tasks T1–T43) are complete for dev_catalog: governed medallion, DQ check-ins, Genie Space, LLM bind, multi-tab app, and security review sign-off in the governance log.

License

  • CursorBricks application & project docs: MIT — Copyright CursorBricks contributors; co-author: Cursor (AI coding agent).
  • Databricks MCP / AI Dev Kit (.cursor/mcp/): Databricks License — see NOTICE and docs/upstream/.

About

Governed travel-marketplace analytics on Databricks — medallion lakehouse, enforceable data quality, Genie for questions that hit real SQL, and an LLM assistant grounded in catalog metadata. Built with Cursor as co-author under a fail-closed security policy. (E2E POC)

Topics

agentic-codingagentic-engineeringagentic-workflowhuman-ai-collaborationhuman-in-the-loop

Resources

Readme
License

Security policy

Security policy
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages