Adding ontology converters

[vmihalovski]

OSI Ontology Converters — Python Project

This PR introduces converters/ontology as a standalone, installable Python project providing bidirectional conversion between ontology formats used across the OSI ecosystem. It adds a complete package: the in-memory OSI ontology model, spec (de)serialisation, a Palantir ingestion layer, three converters, a runnable CLI script, and a test suite.

34 files, ~8.2k lines added. Everything lands under converters/ontology/ — no existing files are modified.

Model layer

• src/osi/model.py — the in-memory OSI ontology model, organised in three layers:

  • ontology — Concept (entity / value / component types), Relationship, Role, RelationshipMultiplicity, Formula, OntologyComponent, OsiOntology
  • semantic — Dataset, DatasetField, Dimension, JoinPath, Metric, DialectExpression / DialectExpressionSet, SemanticModel
  • mapping — ObjectMapping, ReferentMapping, LinkMapping, ConceptMapping, OntologyMapping

  Also includes verbalization parsing (parse_verbalizations) that splits madlib-style relationship text into roles using the dash convention, and FormulaFactory / MappingFormulaFactory extension points so downstream consumers can plug in their own expression parsers for derived_by / requires and mapping expressions.

• src/osi/spec.py — pydantic DTOs mirroring the OSI YAML specification (OsiSpec and friends), used as the serialisation boundary.

• src/osi/__init__.py — a curated public API surface, so consumers import from osi rather than deep sub-paths.

Converters

Converter	Direction	Notes
palantir_to_osi	Palantir ontology → OSI model	maps object types, properties, and datasets; handles many-to-one, many-to-many, and intermediary relations; derives identifying fields and falls back gracefully when none are found; attaches link mappings to concept mappings
osi_to_spec	OSI model → OSI spec YAML	emits non-built-in concepts in dependency order, including ontology-level and concept-level requires
spec_to_osi	OSI spec YAML/JSON → OSI model	completes the round trip; topologically sorts the concept dependency graph before construction

OsiParser (src/osi/parser/) is the entrypoint for reading a spec file (YAML or JSON, UTF-8 pinned for reproducibility) and returning an OsiOntology.

Palantir ingestion

• src/osi/external/palantir/model.py — DTOs for a Palantir export: Ontology, ObjectType, Property, DataSet / DataSetColumn, and the relation hierarchy (ManyToOneRelation, ManyToManyRelation, IntermediaryRelation).
• src/osi/external/palantir/parser/ — PalantirParser accepts a .zip archive or an already-extracted folder, and tolerates both top-level and single-root-directory layouts. It parses the ontology JSON plus a data_sets/ folder of dataset specs, handles both old- and new-style relation encodings, and raises clear errors for ambiguous or missing inputs.

Common utilities

• common/graph.py — topological sort, cycle detection, and a cycle-breaking variant used to order concept and mapping dependencies.
• common/file_utils.py — zip/directory traversal helpers with input validation.
• common/utils.py — identifier and verbalization string normalisation.

Tests

A pytest suite (tests/, 7 files) driven off the flights example:

• test_osi_parser.py — spec → model conversion: document metadata, ontology- and concept-level requires, value-type inheritance, identifiers, relationship multiplicity, ontology mappings, YAML/JSON loading, error handling, formula-factory isolation between parsers, and round-trip structural invariants
• test_palantir_parser.py — every supported export layout (top-level zip, single-root zip, extracted dir, single-root dir, dir wrapping a zip) plus the unsupported/invalid cases (missing path, non-zip file, missing or empty data_sets, ambiguous top-level JSON)
• test_verbalization_parsing.py — prefix/postfix splitting, multi-word decoration, ternary relationships, built-in concept resolution, and error cases
• test_flights_snapshot.py — structure and round-trip YAML snapshots (pytest-snapshot), which also assert the output is deterministic for fixed input
• test_examples_in_sync.py — guards the vendored tests/fixtures/flights.yaml against drift from the canonical examples/flights.yaml. Test inputs are vendored so the suite runs from an sdist/wheel or a subset checkout; this test skips when examples/ isn't present.

Run with pytest from converters/ontology.

Project setup

• pyproject.toml — runtime deps (pydantic, pyyaml), dev extras (pytest, pytest-snapshot, pip-tools), src/ layout, pytest and pyright config; requires Python ≥ 3.11
• requirements.lock — fully reproducible installs, generated from pyproject.toml via pip-compile
• src/osi/ layout — importable as osi after installation
• .gitignore — Python artifacts, virtualenvs, pyenv, VS Code, JetBrains
• README.md — pyenv/virtualenv setup, lock-file regeneration, usage, and script docs

Script

scripts/palantir_to_osi.py — takes a Palantir export (.zip or extracted folder) and writes OSI YAML to stdout, warnings to stderr. Snowflake database and schema names used to qualify table references are configurable via SNOWFLAKE_DATABASE_NAME and SNOWFLAKE_SCHEMA_NAME (both default to PALANTIR).

Not included

No CI wiring — the suite is currently run locally.

[Copilot]

Pull request overview

Introduces a new standalone Python package under converters/ontology that implements OSI ontology converters and supporting parsing/model layers, enabling bidirectional conversion between Palantir exports, an OSI runtime model, and an OSI YAML spec format.

Changes:

• Added Pydantic DTO schema (OsiSpec) plus YAML load/dump helpers for OSI spec serialization.
• Added a runtime ontology + semantic model implementation and converters: Spec ↔ OSI model, Palantir → OSI model.
• Added Palantir ZIP parsing utilities/models and a CLI script + packaging/docs scaffolding for installable usage.

Reviewed changes

Copilot reviewed 16 out of 24 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
converters/ontology/src/osi/spec.py	Defines the Pydantic spec DTOs and YAML load/dump helpers.
converters/ontology/src/osi/model.py	Adds the runtime ontology/semantic/mapping model used by converters.
converters/ontology/src/osi/parser/init.py	Adds an OsiParser entrypoint for reading YAML/JSON into the runtime model.
converters/ontology/src/osi/converter/spec_to_osi/converter.py	Implements Spec → runtime model conversion (incl. mapping-expression resolution).
converters/ontology/src/osi/converter/osi_to_spec/converter.py	Implements runtime model → Spec DTO conversion and mapping-expression rendering.
converters/ontology/src/osi/external/palantir/parser/init.py	Adds Palantir export ZIP parsing (ontology + dataset specs).
converters/ontology/src/osi/external/palantir/model.py	Adds Palantir ontology/data model types used by the parser/converter.
converters/ontology/src/osi/converter/palantir_to_osi/converter.py	Implements Palantir → runtime model conversion (concepts, relationships, mappings, datasets).
converters/ontology/src/osi/common/utils.py	Adds small shared string/name utilities used by converters/parsers.
converters/ontology/src/osi/common/graph.py	Adds topo-sort utilities used for dependency ordering (incl. cycle breaking).
converters/ontology/src/osi/common/file_utils.py	Adds ZIP helpers for reading top-level and dataset JSON files.
converters/ontology/src/osi/init.py	Defines the package’s intended public API surface (osi.*).
converters/ontology/scripts/palantir_to_osi.py	Adds a runnable CLI script for Palantir ZIP → OSI YAML conversion.
converters/ontology/pyproject.toml	Adds Python project packaging/configuration for the new package.
converters/ontology/requirements.lock	Adds a pinned dependency lock file for reproducible installs.
converters/ontology/README.md	Adds setup and usage documentation for the package/script.
converters/ontology/.gitignore	Adds Python-focused ignores for the new project directory.
converters/ontology/src/osi/converter/init.py	Package init for converter modules.
converters/ontology/src/osi/converter/spec_to_osi/init.py	Package init for Spec → OSI converter.
converters/ontology/src/osi/converter/osi_to_spec/init.py	Package init for OSI → Spec converter.
converters/ontology/src/osi/converter/palantir_to_osi/init.py	Package init for Palantir → OSI converter.
converters/ontology/src/osi/external/init.py	Package init for external integrations.
converters/ontology/src/osi/external/palantir/init.py	Package init for Palantir integration.
converters/ontology/src/osi/common/init.py	Package init for common utilities.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 24 changed files in this pull request and generated 5 comments.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 24 changed files in this pull request and generated 9 comments.

[Copilot]

Pull request overview

Copilot reviewed 24 out of 33 changed files in this pull request and generated 3 comments.

[jbonofre]

License headers + CI workflow

Two blocking ASF-convention gaps from the review, with fixes attached.

1. License headers — 21 inline suggestions below add the standard Apache-2.0 header (one-click Apply). The 8 empty __init__.py files need it too but have no suggestable line; the generated requirements.lock, .gitignore, and the tests/snapshots/** files should stay header-less (a header in the round-trip snapshot would break that test — exclude them via RAT instead).

2. CI workflow — every other converter ships .github/workflows/converter-<name>-ci.yml; this PR has none, so the suite never runs on PRs. New file .github/workflows/converter-ontology-ci.yml (mirrors converter-omni-ci.yml, adapted to this converter's pip/requirements.lock setup and 3.11+ floor):

#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.
#

name: Converters Ontology CI

on:
  push:
    branches: [ "main" ]
    paths:
      - 'converters/ontology/**'
      - '.github/workflows/converter-ontology-ci.yml'
  pull_request:
    branches: [ "main" ]
    paths:
      - 'converters/ontology/**'
      - '.github/workflows/converter-ontology-ci.yml'

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.11", "3.12", "3.13", "3.14"]

    steps:
      - name: Checkout project
        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
        with:
          python-version: ${{ matrix.python-version }}

      - name: Install dependencies
        working-directory: converters/ontology
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.lock
          pip install -e ".[dev]"

      - name: Unit Tests
        working-directory: converters/ontology
        run: |
          pytest

I verified the header additions + these exact CI steps in a clean venv: 52 passed. (Couldn't push directly — Allow edits by maintainers is off on this PR.)

[jbonofre]

At least the ASF header is missing.

Also, are you sure about the name? Is it ontology converter or palantir converter?

Can you also rename from OSI to Ossie?

[vmihalovski]

@jbonofre thank you for the detailed review — all points addressed.

• Allow edits by maintainers — enabled.

• OSI → Ossie naming — renamed throughout. The package is now ossie_ontology (was osi) and the distribution apache-ossie-ontology (was ontology), with the module paths and public types following: converter/palantir_to_ossie/, converter/ossie_to_spec/, converter/spec_to_ossie/, and OssieOntology, OssieParser, OssieSpec, PalantirToOssieConverter, OssieToSpecConverter, SpecToOssieConverter. Done with git mv so file history is preserved. The on-disk YAML format contains no osi tokens, so it is unchanged and the fixtures did not move.

• License headers — now on every file in the package. Rather than excluding the snapshots via RAT, I made the header part of the value the snapshot tests assert: tests/conftest.py overrides the snapshot fixture to prepend it, so it applies both when comparing and when regenerating with --snapshot-update, and the tests themselves stay free of boilerplate. The header therefore cannot drift out of sync with the snapshots. # starts a comment in YAML, so flights_roundtrip.yaml is still parseable as an Ossie document.

The one remaining file without a header is the generated requirements.lock, since pip-compile rewrites it and would drop the header — consistent with how the other Python converters treat their uv.lock.

• CI — added .github/workflows/converter-ontology-ci.yml, mirroring converter-omni-ci.yml and adapted to this converter's pip/requirements.lock setup, over a 3.11–3.14 matrix. I verified the install steps and the suite locally: 52 tests pass.

• On the ontology package name — it's deliberately format-neutral rather than named after any one source. The package holds several converters already (Palantir → Ossie, Ossie ↔ Spec YAML) and we expect to add more, most likely OWL, so naming it after a single format would age badly. As apache-ossie-ontology under converters/ontology it also stays consistent with the other converter packages.

Could you take another look when you get a chance? Thanks again.

[jbonofre]

It's a great contribution and now very solid. Thanks!

[vmihalovski]

@jbonofre, thank you for the approval. Could you please let me know what I need to do to merge this PR? I don’t see a merge button, and I understand that merge permissions may be limited. Would you be able to help with the merge?

Thank you in advance for your help.

[jbonofre]

I will merge. I just give time for the other PPMC members to chime in if they want.

[kurtStirewalt]

@jbonofre , just checking back on this, as it's been a couple of weeks. Could you please go ahead and merge this PR?
Thanks

[vmihalovski]

@jbonofre We received a new Palantir export with slightly different attributes, so I made a few changes to our converter to accommodate the new format.

Palantir parser & converter: status policy, export-format tolerance, datasets optional

• Status handling is now a policy. The hardcoded active() or endorsed() or … checks scattered through the converter became six class attributes read through allowed_*_statuses() hooks. Defaults unchanged — the win is that a draft ontology (in one export, 510 of 552 object types are experimental) can widen them in one line per policy instead of forking the converter.
• Subtype detection moved from the model to the converter (Ontology.subtypes_relations() → _subtype_relations()). Palantir has no subtype construct — it's inferred from a M:1 relation pairing primary key to primary key — so it's a converter interpretation, and it carried its own hardcoded copy of the status rule. The converter was its only caller.
• Datasource identifiers: datasourceRid or rid. Newer (v1/v2/v3) exports use rid; the parser rejected them outright. Reads go through DATASOURCE_RID_KEYS, so another spelling is one entry.
• Primary-key columns fall back to a case-insensitive match. Exports disagree on case between primaryKeyMapping (LOCNO) and the property (locno); unquoted Snowflake identifiers are case-insensitive anyway, and losing the identifying column drops the whole concept mapping. Exact match still wins; a genuinely missing column still warns.
• Missing datasets warn instead of raising. SDK-extracted exports ship this way, and the ontology JSON alone yields a structurally complete model — just one with no mappings, so every query returns nothing. Worth a loud warning, not a refusal to parse.

Tests — new test_palantir_converter.py (17) plus a datasource section in test_palantir_parser.py (5), each building the smallest real-shaped export and running it through the actual parser.

Would you mind reviewing my changes, approving this PR, and merging it afterwards?
Thank you in advance for your help.

[jbonofre]

Reviewed the Aug 18 changes (status policy / export-format tolerance). The structure is solid and the new test_palantir_converter.py coverage is welcome. I did find a set of issues in the new converter paths, and I verified each one against the actual parser/converter rather than by reading, so the repro steps below should all reproduce.

I think we produce wrong output or abort a conversion on export shapes this PR is specifically meant to handle:

1. Subtype relations are emitted twice — as extends and again as an ordinary relationship. Reproduces with this PR's own test_primary_key_to_primary_key_relation_becomes_inheritance fixture.
2. referent_mapping order is non-deterministic for composite keys, which makes flights_roundtrip.yaml flaky.
3. assert parent is not None is reachable from an ordinary export and aborts the run.
4. A relation whose name collides with a property kills the whole conversion.
5. DataType.to_type()'s catch-all silently types attachments, media references and vectors as Integer.

[jbonofre]

I did a pass on the August 18 changes. The changes introduce 5 things worth to fix. Can you please take a look?

[vmihalovski]

@jbonofre Thank you for your review. Please take a look at the summary below for an overview of the changes made in this commit. Would you mind taking another look and, if everything looks good, approving and merging the PR?

Palantir converter: correctness fixes

Fixes a set of defects in the Palantir → Ossie conversion paths and the shared model:
wrong output on export shapes the converter is meant to handle, and aborts on shapes it
should tolerate. Each fix comes with a test that fails without it.

Tests: 74 → 143. Suite passes under pytest and pytest -O.

Inheritance

• A PK-to-PK M:1 relation was emitted twice, as an extends and as an ordinary
  relationship. _convert_concepts now returns the guids it consumed as inheritance and
  _convert_relationships skips them. This is narrower than _subtype_relations(): a
  subtype edge dropped to break a cycle stays an ordinary relation, and only
  _convert_concepts knows which those are.
• A relation mapping only part of the parent's primary key counted as inheritance and
  then died in _convert_mappings with a bare KeyError naming no property, dataset or
  relation. Inheritance now requires the parent's whole key.
• A supertype excluded by the object-type status policy while its relation passed the
  subtype-relation policy hit assert parent is not None — reachable with stock
  defaults, and under -O it built Concept(extends=[None]) and failed later on a
  None. Now a ValueError naming the two policies that disagree.

Concept mappings and dataset fields

• referent_mapping order was non-deterministic for composite keys, because
  primary_keys() is a set of identity-hashed properties — 6 distinct orderings of a
  3-column key over 60 conversions in one process. Both identifier_props branches now
  sort by readable_id, as identify_by already did.
• A link was attached to another dataset's ConceptMapping when the right one had been
  dropped, producing a mapping that joins two unrelated tables under one identity. A
  dataset match is now required; without one the link is dropped with the existing
  warning. _references_dataset also inspects om.expression, not only
  om.referent_mappings.
• Field names were not sanitized: sanitize_identifier was unused while
  _normalize_field_name only replaced -, so a column named net amt failed the
  identifier pattern on reparse and was silently demoted to a Formula instead of
  resolving to the DatasetField. Wired up, with the field's SQL expression computed
  separately and quoted so it keeps naming the real column.

Relations

• A relation whose name collided with a property's killed the whole conversion, since
  add_relationship raises on a duplicate — a common export shape, where the link and
  the foreign-key column backing it share a name. Added
  _add_relationship_if_name_free, used by all three relation kinds (M:1, M:M,
  intermediary): the link is dropped with a warning and the run continues.
• Added the first coverage for the M:M and intermediary paths, which had no test fixtures
  at all.

Type mapping and export tolerance

• DataType.to_type()'s catch-all typed several non-numeric types as Integer:

  Palantir type	before	after
  ATTACHMENT, MEDIA_REFERENCE, CIPHER_TEXT	Integer	String
  STRUCT, VECTOR	Integer	String
  ANY	Integer	Any
  LONG, SHORT	Integer	Integer

• One unrecognized column type (BINARY, MAP<…>) aborted the conversion, over a value
  not serialized downstream, while a missing type on the same line already fell back to
  String. Now warned and carried.

• A datasetSchema entry with no name parsed into DataSetColumn(None, …) and crashed
  much later with AttributeError: 'NoneType' object has no attribute 'replace'. Skipped
  at parse with a warning; routing through norm() also catches whitespace-only names.

Names

• to_verbalization_string reported any name starting with a digit as containing
  unsupported symbols it does not have. The warning now precedes the leading-digit
  substitution; return values are unchanged.
• to_pascal_case could produce digit-leading concept names (3m corp → 3mCorp),
  illegal to the formula lexer. Now spelt out the way to_verbalization_string already
  does, keeping the rest pascal-cased → ThreeMCorp.

Model

• lookup_concept mutated the ontology, creating and adding a builtin as a side effect,
  so lookup_concept(name) is None was not the existence check it reads as. Split into a
  pure lookup_concept and a mutating ensure_builtin_concept. An object type whose
  PascalCase name is a builtin — which used to take that name over, so every column of
  that builtin type resolved to the entity — is now reported as a concept-name collision.
• Role.__eq__/__hash__ ignored position, so the two roles of an unnamed
  self-relationship compared equal: sibling returned the role it was asked from, and
  both collapsed into one entry of any set keyed by Role.
• _topological_sort stalled on an edge naming a node outside nodes exactly as on a
  real cycle, so a dangling extends was reported as a cycle and the accurate "Subtype
  'X' is not declared" message was unreachable. Those edges are now skipped, as
  _topological_sort_break_cycles already skipped them.
• _convert_ontology_concepts rescanned a freshly rebuilt relationship list once per
  concept; grouped by container in one pass, output byte-identical.

Behaviour changes to be aware of

1. lookup_concept no longer creates builtins — callers wanting that need
   ensure_builtin_concept. (osi-lab checked: unaffected.)
2. to_pascal_case renames digit-leading concepts, so YAML previously generated for such
   a model will diff.
3. An object type whose PascalCase name is a builtin now raises instead of silently
   merging into it.
4. A link whose dataset has no ConceptMapping is dropped with a warning instead of being
   attached to another dataset's mapping.
5. Dataset field names are fully sanitized, and non-identifier column names are quoted in
   the emitted SQL rather than reproduced bare.
6. Attachment / media-reference / cipher-text / struct / vector properties are typed
   String and ANY is typed Any, where all six were Integer.