Skip to content

feat(data-weaver): child place resolution - #447

Merged
nick-nlb merged 7 commits into
datacommonsorg:mainfrom
nick-nlb:dw-child-place-resolution
Aug 24, 2026
Merged

feat(data-weaver): child place resolution#447
nick-nlb merged 7 commits into
datacommonsorg:mainfrom
nick-nlb:dw-child-place-resolution

Conversation

@nick-nlb

@nick-nlb nick-nlb commented Aug 22, 2026

Copy link
Copy Markdown
Collaborator

Issue

This PR addresses tracker lines 10, 11, 16 and 19.

Description

The goal of this PR is to update place resolution across the query execution pipeline, store and front-end in order to support sub-region and child place retrieval and multi-series chart visualizations.

An important note is that while this PR does update skills and agent directions in order to improve the results of searches involvin child places, its primary goal is to create the update data fetching and the end-to-end pipeline to support child searching and display. We will continue to improve the skills in future PRs.

Implementation

  • Updated agent instructions in order to have the agent able to understand whether a query should be a child place query.
  • Fixed store structure and query pipeline so that the connection between the actual place resolved and the stat var is preserved (this addresses the otherwise very frequent "no data" issues). (Addressing items 10 and 16)
  • Implemented child place hierarchy retrieval using the Data Commons /v2/node API (<-containedInPlace+) to fetch all sub-entities for a given parent region. As part of this, introduced place normalization to Data Commons place types for reliable child fetching.
  • Extended time-series observation fetching to support multi-entity requests across child places.
  • Implemented composite scope key generation to keep canvas cards, history nodes, and the store synchronized.
  • Centralized card title formatting and removed fragile title parsing. (Addressing item 19)

Notes

Upcoming items, to be implemented in separate PRs:

  • Implementation of choropleth and ranking tiles (both better suited to handle large numbers of child places).
  • Limits to the number of series that display on a line chart (e.g. "Displaying top x…").
  • General improvements to line charts and tooltips.
  • Refactoring of query stream handling (right now, this function is very long, and longer still with this PR).

Tests

Added unit tests for place type normalization, scope key derivation, and title formatting.

Testing

"How access to electricity has changed across countries in Africa"

On main branch

  • In main, the agent does not recognize the "child resolution" nature of the query and either produces unsatisfactory results or, worse, results in "No Data" because of a mismatch between the entities and stat vars.
  • The reason for the "No Data" is that in main, the model looks for data for the primary entity ("Africa"), does not find it, and falls back to "Earth" where it finds data (and therefore stat vars). However, it loses the connection between the stat var and the actual entity found, and attempts to apply it to Africa where there is no data. This core issue is fixed (along with the improvement to perform a child place search rather than fall back to Earth).

In this PR

The search will resolve to child places, and display all countries on a line chart.

  • Note: the line chart is not the best tool for this type of data: this will be addressed in upcoming PRs.
  • The search itself, being deterministic, does not always perform a child search. We can harden this in subsequent PRs.

…h this, it introduces refactoring of and fixes to composite scope key derivation and updates to card generation to render multi-series for child entities.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for sub-region and child-place queries (e.g., querying counties within a state) by fetching child entities and their observations, normalizing place types, and resolving deterministic scope keys. Key changes include new helper functions for formatting card titles, normalizing place types, and managing composite scope keys, alongside updated LLM instructions for data discovery. The review feedback highlights a critical bug in render_result_html.ts where unnormalized child place types in markdown links would cause frontend resolution failures, and suggests defensive programming improvements in scope_key.ts and observations.ts to filter out nullish entity DCIDs and invalid observation dates before sorting.

Comment thread dataweaver/apps/web/src/server/steps/render_result_html.ts
Comment thread dataweaver/apps/web/src/server/steps/render_result_html.ts
Comment thread dataweaver/apps/web/src/functions/scope_key.ts
Comment thread dataweaver/apps/web/src/server/steps/observations.ts
@nick-nlb
nick-nlb marked this pull request as ready for review August 22, 2026 20:49
…dence over defaults when given (i.e., counties in the USA)

@beets beets left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, and nice to see more tests! Left comments, but many of these aren't gating -- more for follow ons. Can be captured in TODOs.

Comment thread dataweaver/apps/web/src/server/config/skills/data_discovery.md
Comment thread dataweaver/apps/web/src/server/config/skills/data_discovery.md
Comment thread dataweaver/apps/web/src/app/api/query/route.ts
Comment thread dataweaver/apps/web/src/functions/normalize_place_type.ts
@nick-nlb
nick-nlb merged commit d020630 into datacommonsorg:main Aug 24, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants