Skip to content

application-profiles.md

Latest commit

 

History

History
287 lines (238 loc) · 15.4 KB

application-profiles.md

File metadata and controls

287 lines (238 loc) · 15.4 KB
Raw

Application profiles

The application profile is how an institution declares what it wants generated and how. It is one file, application-profile.json, conforming to ../schemas/application-profile.schema.json. It names the target schema, the controlled vocabularies in priority order, the house description style, the fields that must be right for a record to pass, the review threshold the loop optimizes toward, and the model to use for each role. From it, two things follow: the visible {{...}} placeholders in the generation prompt are filled, and the loop and review stages get the gate, the threshold, and the model choices they run on.

The profile lives in the institution's adjacent workspace, never in this shared repo. This doc describes the artifact; it does not ship one. The skills/build-application-profile skill drafts a profile from the works plus the institution's context, and the institution owns and edits it thereafter. The schema is closed (additionalProperties: false): add behavior the schema does not cover through the local overrides/ directory referenced by overrides_path, not by inventing new profile keys. See adjacent-workspace.md.

What is required, and what is optional

A minimal valid profile sets the seven required fields:

  • institution_name
  • institution_type
  • target_schema
  • controlled_vocabularies
  • required_fields
  • review_threshold (with per_record_pass_rate)
  • model (with at least generation)

The optional fields are description_style, collection_hierarchy, and overrides_path. Each does real work even though it is not mandatory; the sections below say what consumes each.

Three regions: what the profile feeds

It helps to read the profile as three regions, because the fields flow to three different places and conflating them implies coupling the schema does not have.

  1. Configuration placeholders that fill the visible {{...}} blanks in the generation prompt.
  2. Provenance placeholders in that same prompt that the run stamps, not the profile.
  3. Profile fields that drive the loop and review stages and never appear in the prompt text at all.

Region 1: the configuration placeholders

The generation prompt at ../prompts/system/metadata-generation.v1.md is deliberately readable and editable, and its {{...}} placeholders are the only configuration in it. These are filled directly from the profile:

Profile field Prompt placeholder What the generator does with it
institution_name {{INSTITUTION_NAME}} Frames whose collection this is.
institution_type {{INSTITUTION_TYPE}} Frames the kind of materials and conventions in play.
target_schema {{TARGET_SCHEMA}} The descriptive schema the record targets.
controlled_vocabularies {{CONTROLLED_VOCABULARIES}} Rendered as the priority-ordered vocabulary list the model proposes terms from.
description_style.guidance {{DESCRIPTION_STYLE}} The house style for the description field.
description_style.examples_path {{HOUSE_STYLE_EXAMPLES}} Path to exemplar finished records for few-shot calibration.
model.generation {{MODEL_ID}} Recorded in provenance and the disclosure note as the model that produced the first pass.

Region 2: the provenance placeholders the run stamps

Two placeholders in the prompt are not declared in the profile and are filled by the run, not by configuration:

  • {{PROMPT_VERSION}} is the version id of the prompt itself, for example system/metadata-generation@1.0.0. It is stamped from the prompt that actually ran so every record is reproducible.
  • {{DATE}} is the run date.

Both land in provenance and the disclosure_note so a record records what produced it and when. The profile supplies the configuration placeholders; the run supplies these two. Keeping the distinction clear is what lets the disclosure note mean something: it reports the exact prompt version and date, which are properties of the run, while {{MODEL_ID}} carries through from the profile's chosen generation model.

Region 3: profile fields that drive the loop and review, not the prompt

The remaining fields never appear as {{...}} blanks. They configure the gate, the loop, and the composition of a run:

  • required_fields defines the per-record pass gate (see below).
  • review_threshold is the loop's optimization target (see below).
  • model.orchestration and model.local_only_collections configure the loop's roles and the local-only constraint (see below).
  • collection_hierarchy is supplied to the generation stage as evidence so works are described in context, captured in the record's collection_context rather than injected as a prompt blank. See collection-context.md.
  • overrides_path points the run at the local prompt, skill, and agent customizations that take precedence over the shared repo.

Target schema

target_schema names the descriptive standard the record targets, for example Dublin Core, Dublin Core + VRA Core for visual works, or PBCore for audiovisual. It fills {{TARGET_SCHEMA}} and is written into every record's target_schema. The prompt's documented default is Dublin Core, with VRA Core elements for visual works and PBCore elements for audiovisual. The export stage adapts the final write to the institution's platform regardless of the schema; see architecture.md and the standard-by-content-type guidance in the initiative's metadata-standards reference.

Controlled vocabularies, including community and local

controlled_vocabularies is an array, and three things about it matter.

Array order is the priority order. The generator proposes terms preferring earlier vocabularies in the list. The rendered list fills {{CONTROLLED_VOCABULARIES}}. The prompt's documented default order is LCSH and FAST for subjects, Getty AAT for object type, material, technique, and style, Getty TGN for places, LCNAF and VIAF for named persons and organizations, LCGFT for genre and form, and Homosaurus where relevant for LGBTQ+ subjects.

applies_to binds a vocabulary to the fields it governs. A profile says, for example, that LCSH and FAST apply to subjects while Getty AAT applies to medium_technique, so the generator does not reach for a subject vocabulary on a material field.

Community and local vocabularies belong here alongside the national authorities. The name may be a national authority (LCSH, FAST, Getty AAT, Getty TGN, LCNAF, VIAF, LCGFT), a community vocabulary such as Homosaurus for LGBTQ+ subjects, or a local scheme id for an institution's own controlled list. The scope_note is where an institution records when to prefer one vocabulary over another, including community-specific guidance. Community vocabularies are part of how the toolkit answers the cultural-bias risk; see responsible-description.md.

The endpoint field has a second consumer. It is the linked-data or API endpoint used by the deterministic authority-resolution stage (stage 5), which is where needs_verification candidates become confirmed URIs. The prompt receives the labels and the priority order; the authority stage receives the endpoints. The LLM proposes controlled values, the authoritative lookup disposes, and the human confirms. See authorities.md.

House style

description_style is the optional house-style block, tuned to local practice, and it fills two placeholders. description_style.guidance fills {{DESCRIPTION_STYLE}}: the documented default is factual, concise, present tense, describing what is visibly present without interpreting intent or speculating about meaning. description_style.examples_path fills {{HOUSE_STYLE_EXAMPLES}}: a path in the adjacent workspace to exemplar finished records the generator uses for few-shot calibration of length and register. The optional max_length_words caps description length. Pointing examples_path at the institution's own approved records is the most direct way to make generated descriptions read in the house voice; the skills/tune-house-style skill helps build that exemplar set.

Required fields, and why each one raises the bar multiplicatively

required_fields lists the fields that must be accepted with no edit for a record to pass the per-record gate. It does not appear in the prompt; it configures the gate the loop measures against.

The gate is strict by construction, and the cost of each required field is multiplicative, not additive. The reasoning is worked out in the-improvement-loop.md: if a record has ten required fields and each is independently correct 95 percent of the time, the whole record needs no edit only about 60 percent of the time (0.95 raised to the tenth power). To clear a 90 percent per-record bar across ten fields, each field has to be right about 99 percent of the time. Adding a required field multiplies one more probability below 1 into that product, so it is never free.

Two definitions sharpen what "passes" means for the gate:

  • A required scalar field passes when its value is accepted with no edit.
  • A required array field (subjects, genre_form, depicted_entities) is held to the harder standard: it passes only when every generated value in it is accepted and the cataloguer added no missing value. One edited, rejected, or missing value fails the field, and a failed required field fails the record.

The schema's instruction is therefore to keep required_fields honest and minimal. Make a field required because the institution genuinely will not ship a record without it, not by default. Review verdicts attach to each value's stable value_id, never to an array position, so per-field attribution stays correct even when a prompt change reorders an array between rounds.

Review threshold

review_threshold.per_record_pass_rate is the single number the institution sets: the share of generated records that need no human modification. Oregon State, for example, uses roughly 90 percent (0.90). The optional rationale records the policy behind it, for example that below the line the institution reviews the entire dataset by hand.

This number is the institution's own review-policy gate and the loop's optimization target. It is not a contractual accuracy guarantee from Notch8. The pilot promises the method and the transparency, not a number. The loop raises the per-record rate above this threshold on a sample, confirms it on a fresh held-out sample, and only then scales the large run. See the-improvement-loop.md.

Model choice

model selects which model plays each role; model is a parameter, not a fixed dependency.

  • model.generation is the default generation model id, for example claude-sonnet-4-6 or a local endpoint id. It fills {{MODEL_ID}} and is recorded in provenance. The cost-effective model does the volume generation.
  • model.orchestration is the frontier model the loop uses to orchestrate runs and author prompt, skill, and agent refinements, for example claude-opus-4-8. Frontier capability is confined to the small sample, which is what keeps the loop economical.
  • model.local_only_collections lists collections that must use a local model with nothing sent to an external API. For those collections the run uses a local model and sets provenance.local_only to true, which is the auditable proof that no content left the institution's infrastructure. This is the path for sensitive or rights-restricted material.

See models-and-cost.md.

Collection hierarchy and overrides

collection_hierarchy optionally declares the collection tree so the generation stage receives parent context as evidence and the record can describe the work as part of a whole. It is supplied to generation, not rendered as a prompt blank, and it is detailed in collection-context.md.

overrides_path points at the local prompt, skill, and agent customizations in the adjacent workspace. When an asset exists in both the local overrides/ and this repo, the local one wins. This is how an institution tailors behavior, and how the generalizable-versus-domain split is enforced in practice: a change that would help an institution describing entirely different works contributes upstream, while a change that encodes this collection's scope, house style, local vocabulary, or donor context stays in overrides/. See the-improvement-loop.md and ../CONTRIBUTING.md.

A filled profile

A complete, valid profile, filled and visible:

{
  "institution_name": "Example State University Special Collections",
  "institution_type": "university special collections library",
  "target_schema": "Dublin Core + VRA Core",
  "controlled_vocabularies": [
    {
      "name": "LCSH",
      "applies_to": ["subjects"],
      "endpoint": "https://id.loc.gov/authorities/subjects",
      "scope_note": "Primary subject vocabulary."
    },
    {
      "name": "FAST",
      "applies_to": ["subjects"],
      "endpoint": "https://id.worldcat.org/fast",
      "scope_note": "Use for faceted subjects where an LCSH heading is unwieldy."
    },
    {
      "name": "Getty AAT",
      "applies_to": ["genre_form", "medium_technique"],
      "endpoint": "https://www.getty.edu/research/tools/vocabularies/aat",
      "scope_note": "Object type, material, technique, and style."
    },
    {
      "name": "Getty TGN",
      "applies_to": ["spatial"],
      "endpoint": "https://www.getty.edu/research/tools/vocabularies/tgn",
      "scope_note": "Places."
    },
    {
      "name": "LCNAF",
      "applies_to": ["creator", "depicted_entities"],
      "endpoint": "https://id.loc.gov/authorities/names",
      "scope_note": "Named persons and organizations."
    },
    {
      "name": "Homosaurus",
      "applies_to": ["subjects"],
      "endpoint": "https://homosaurus.org",
      "scope_note": "Prefer for LGBTQ+ subjects; interoperable with LCSH."
    },
    {
      "name": "esu-local-formats",
      "applies_to": ["genre_form"],
      "endpoint": null,
      "scope_note": "Local format scheme for departmental record types not covered by LCGFT."
    }
  ],
  "description_style": {
    "guidance": "Factual, concise, present tense. One to two sentences. Describe what is visibly present; do not interpret intent or speculate about meaning.",
    "examples_path": "context/exemplar-records/",
    "max_length_words": 60
  },
  "required_fields": ["title", "description", "type", "date", "subjects"],
  "review_threshold": {
    "per_record_pass_rate": 0.90,
    "rationale": "Below 90 percent, library policy is full-dataset human review."
  },
  "model": {
    "generation": "claude-sonnet-4-6",
    "orchestration": "claude-opus-4-8",
    "local_only_collections": ["student-medical-records-ca-1950-1980"]
  },
  "overrides_path": "overrides/"
}

This profile declares a visual-works schema, six national and community vocabularies plus one local scheme in priority order with their fields and endpoints, a concise house style with an exemplar path, five honestly required fields, a 90 percent per-record gate, a cost-effective generation model with a frontier orchestrator, one collection pinned to local-only generation, and the local overrides directory. Reading it, you can see exactly which blanks the prompt will receive and which fields the loop and review stages will run on.