From cf8bbe2cb80dfe1eaa8862c77252063adaa6a228 Mon Sep 17 00:00:00 2001 From: jamilxt Date: Wed, 23 Sep 2026 18:37:56 +0000 Subject: [PATCH 1/2] Add design doc: Jev-based Micrometer metric triage module --- docs/design-micrometer-metric-triage.md | 125 ++++++++++++++++++++++++ 1 file changed, 125 insertions(+) create mode 100644 docs/design-micrometer-metric-triage.md diff --git a/docs/design-micrometer-metric-triage.md b/docs/design-micrometer-metric-triage.md new file mode 100644 index 0000000..01ba151 --- /dev/null +++ b/docs/design-micrometer-metric-triage.md @@ -0,0 +1,125 @@ +# Design: typesafe-ai-java-micrometer + +Jev-style metric triage for JVM applications, in the spirit of ishantanu/jevmetrics +(Go OTel Collector processor) but in-process for Micrometer/`MeterRegistry`, built on +`typesafe-ai-java`. New Maven module `typesafe-ai-java-micrometer` in the existing repo. + +## 1. Problem + +jevmetrics sits in an OTel Collector and asks Jev whether a metric is worth keeping: +4 questions (relevance, redundancy, keep, recommended action), deterministic local +policy on the answers, cache by metric identity, `annotate` mode first and `reduce` +(keep-score filtering) second. Nothing equivalent exists for the JVM. Micrometer is the +natural in-process analog: every Spring Boot app already has a `MeterRegistry`, and the +`Meter.Id` (name, tags, unit, baseUnit, description) carries the same metadata evidence +jevmetrics feeds to the model. + +## 2. Goal / non-goals + +Goal: a library that assesses an app's meters through Jev and either (annotate mode) +emits companion `jev.metric.*` gauges for review, or (reduce mode) denies meter +registration below a keep threshold, with an explicit protection allowlist. + +Non-goals (match jevmetrics honesty): +- No downsampling, no aggregation, no dimension reduction. Filtering = deny/allow a + whole meter identity, nothing else. +- No fleet-level telemetry control. In-process only; it affects this app's registry. + Complementary to a Collector-level processor like jevmetrics, not a replacement. +- No model training, no re-scoring of values. Only meter metadata is assessed. + +## 3. Module placement + +`typesafe-ai-java-micrometer`, sibling of core / kotlin / spring-ai / starter: +- `io.micrometer:micrometer-core` as `provided` scope (do not force it on consumers). +- Depends on `typesafe-ai-java-core` only. No Spring inside this module: keep it the + zero-Spring lightweight piece, consistent with our positioning vs the community SDK. +- Spring Boot auto-config lives in the existing `spring-boot-starter` module + (new auto-configuration class + properties, registered in + `AutoConfiguration.imports`). + +## 4. Core API (micrometer module) + +``` +JevMetricAssessor // orchestrates: extract -> dedupe -> evaluate -> cache +JevMeterAssessment // record: Meter.Id coordinates, 4 answer values, timestamp +JevMeterFilter implements MeterFilter // the Micrometer entry point +JevAssessmentCache // bounded LRU + TTL, keyed by stable meter identity +JevProtectedMetrics // explicit allowlist, bypasses inference entirely +``` + +Key decisions: + +- **Identity = normalized `Meter.Id`.** Name + tag key/value pairs sorted, plus unit. + Stable string key (like jevmetrics' cache identity) so the same meter never triggers + a second inference call. +- **Jev call: all four questions in ONE evaluate** (per API reference; ~500-800 input + tokens, ~0.6-2.4s via gateway). Relevance and redundancy are informational; + `keep` noul drives policy; recommended `action` is exposed but never overrides + the keep-score threshold. +- **Async by design.** The filter must never block meter registration on a network + call. On first sight of a new meter: admit it (annotate) or apply cached/default + policy, enqueue the assessment, background worker calls Jev, cache updated. + Subsequent batches/registrations use the cached decision. +- **annotate mode**: original meter always registered. Companion gauges + `jev.metric.relevance`, `jev.metric.redundancy`, `jev.metric.keep_probability` + registered under the original meter's tags plus `metric.name` and `jev.model` + tags, mirroring jevmetrics. `jev.metric.recommended_action` as a gauge with an + `action` tag (keep=1). Protected meters get no companions. +- **reduce mode**: `MeterFilter.reply(deny)` when cached keepProbability < + threshold. Before the first assessment lands, default is ADMIT (fail-open), so a + Jev outage can never blackhole telemetry. Protection allowlist always wins. +- **Failure policy**: any Jev error → log at WARN once per identity, meter stays + admitted, retry with exponential backoff. Never throw out of `accept()`. + +## 5. Spring Boot auto-config (starter module) + +``` +typesafe.jev.metrics.enabled (default true when a key exists) +typesafe.jev.metrics.mode annotate | reduce (default annotate) +typesafe.jev.metrics.keep-threshold 0.0-1.0, default 0.5 (reduce mode only) +typesafe.jev.metrics.protected-names list of exact meter names, always admitted +typesafe.jev.metrics.cache-max LRU size, default 1000 +typesafe.jev.metrics.cache-ttl default 24h +typesafe.jev.metrics.worker-threads default 1 +``` + +Backs off with the same `TypeSafeApiKeyCondition` pattern as the existing client +bean (property OR `TYPESAFE_API_KEY` env), `@ConditionalOnMissingBean` on the filter +so user beans win. `ApplicationContextRunner` tests for both modes and for backoff. + +## 6. Testing + +Mirror jevmetrics' test list, mapped to Micrometer, on the existing fake-transport +pattern (no key, no network): +- response validation: malformed/missing keep answer -> failure path, meter admitted +- mode normalization: unknown mode string -> annotate + WARN +- policy: threshold boundary (== threshold keeps), protected-name bypass +- payload preservation: counter/gauge/timer/long-task-timer all pass through + untouched in annotate mode; companion gauges appear with correct tags +- cache: identity dedupe (two `Meter.Id`s differing only in tag ORDER share one + entry), TTL expiry, eviction at max size +- queue saturation: unbounded burst of new meters -> bounded work queue, oldest + dropped, no OOM, no registration blocking +- failure recovery: transport throws -> WARN once, backoff, later success caches +- auto-config: runner tests for both modes, backoff without key, user bean wins +- Surefire 3.2.5 pinned in the module pom; assert on "Tests run" counts, not exit code + +Live smoke test env-gated (`TYPESAFE_API_KEY` + `assumeTrue`), same as core. + +## 7. Milestones + +1. M1: micrometer module - identity, cache, assessor, filter, annotate mode, tests +2. M2: reduce mode + protection rules + policy tests +3. M3: starter auto-config + runner tests + README section +4. M4: examples app + "I built jevmetrics-style triage for Spring Boot" article arc + (gate through check_article_safety.py like every article) + +## 8. Open questions + +- Version line: this is a new module; propose shipping inside 0.3.0 (module addition + is minor, not patch). +- Optional second target later: an OpenTelemetry Java SDK extension for non-Micrometer + apps; explicitly out of scope for M1-M3. +- Naming: `jev.metric.*` companion gauge names copied from jevmetrics for ecosystem + recognizability; alternative is a `typesafe.metric.*` prefix. Default proposal: + keep `jev.metric.*` (the model is Jev, regardless of serving backend). From 6871afcd2513d95f44655fd14331a34efbe0a203 Mon Sep 17 00:00:00 2001 From: jamilxt Date: Wed, 23 Sep 2026 18:44:54 +0000 Subject: [PATCH 2/2] Revise design after critique: review-first repositioning, name-level identity, reduce-mode honesty --- docs/design-micrometer-metric-triage.md | 207 +++++++++++++++--------- 1 file changed, 132 insertions(+), 75 deletions(-) diff --git a/docs/design-micrometer-metric-triage.md b/docs/design-micrometer-metric-triage.md index 01ba151..65f9ae5 100644 --- a/docs/design-micrometer-metric-triage.md +++ b/docs/design-micrometer-metric-triage.md @@ -1,31 +1,59 @@ # Design: typesafe-ai-java-micrometer -Jev-style metric triage for JVM applications, in the spirit of ishantanu/jevmetrics -(Go OTel Collector processor) but in-process for Micrometer/`MeterRegistry`, built on -`typesafe-ai-java`. New Maven module `typesafe-ai-java-micrometer` in the existing repo. +Jev-powered instrumentation review for JVM applications, in the spirit of +ishantanu/jevmetrics (Go OTel Collector processor) but in-process for +Micrometer/`MeterRegistry`, built on `typesafe-ai-java`. New Maven module +`typesafe-ai-java-micrometer` in the existing repo. + +## 0. Repositioning after critique (Sep 23, 2026) + +The original draft framed this as a metric-storage cost tool. That framing is weak +for an in-process library and the critique stands; the framing is now: + +**Primary value: instrumentation review.** The model reads meter metadata and +surfaces dead-looking, redundant-looking, or badly-described meters as a REPORT. +The artifact is a lint for metrics, consumed at review time, not a pipeline that +saves storage bills. + +**Secondary value (opt-in experiment): reduce mode.** Retained because it is the +jevmetrics-parity feature and worth evaluating, but the docs must state its +structural limits (see 4). It is not the headline. + +What we are NOT claiming, ever: downsampling, aggregation, fleet-level cost +savings, or that a model judgment replaces usage-based retention signals. ## 1. Problem jevmetrics sits in an OTel Collector and asks Jev whether a metric is worth keeping: 4 questions (relevance, redundancy, keep, recommended action), deterministic local -policy on the answers, cache by metric identity, `annotate` mode first and `reduce` -(keep-score filtering) second. Nothing equivalent exists for the JVM. Micrometer is the -natural in-process analog: every Spring Boot app already has a `MeterRegistry`, and the -`Meter.Id` (name, tags, unit, baseUnit, description) carries the same metadata evidence -jevmetrics feeds to the model. +policy on the answers, cache by metric identity. Nothing equivalent exists for the +JVM. Micrometer is the natural in-process analog: every Spring Boot app already has +a `MeterRegistry`, and `Meter.Id` (name, tags, unit, baseUnit, description) carries +the metadata evidence the model can judge. + +The honest gap: usage-based systems (Grafana Adaptive Metrics) decide from what is +actually QUERIED, which is stronger evidence than any static-metadata judgment. +Metadata inference is a prior. It is most useful exactly where usage signals do not +exist yet: a brand-new service whose meters nobody has queried or named carefully. +Position the tool there, not against Adaptive Metrics. ## 2. Goal / non-goals -Goal: a library that assesses an app's meters through Jev and either (annotate mode) -emits companion `jev.metric.*` gauges for review, or (reduce mode) denies meter -registration below a keep threshold, with an explicit protection allowlist. +Goal (primary): a review/lint library. On startup (and on demand), assess the app's +meter names through Jev, cache the verdicts, and expose them as (a) companion +`jev.metric.*` gauges for live inspection, (b) a structured report (log + JSON file) +listing meters worth a human look, with the model's reasoning. + +Goal (secondary): `reduce` mode, a `MeterFilter` denying registration below a keep +threshold, protection allowlist, explicitly documented as experimental. -Non-goals (match jevmetrics honesty): -- No downsampling, no aggregation, no dimension reduction. Filtering = deny/allow a - whole meter identity, nothing else. -- No fleet-level telemetry control. In-process only; it affects this app's registry. - Complementary to a Collector-level processor like jevmetrics, not a replacement. -- No model training, no re-scoring of values. Only meter metadata is assessed. +Non-goals: +- No downsampling, no aggregation, no dimension reduction. A verdict applies to a + whole meter name, nothing else. +- No fleet-level telemetry control; in-process only. Complementary to a + Collector-level processor like jevmetrics, not a replacement. +- No model training, no re-scoring of values, no usage tracking. +- No claim that metadata inference beats usage-based retention where usage exists. ## 3. Module placement @@ -37,89 +65,118 @@ Non-goals (match jevmetrics honesty): (new auto-configuration class + properties, registered in `AutoConfiguration.imports`). -## 4. Core API (micrometer module) +## 4. Identity and the cardinality trap + +**Assessment identity = meter NAME only (default), tags excluded.** Critique point: +keying by full `Meter.Id` makes the tool spend Jev calls proportional to tag +cardinality, exactly the disease it exists to treat. One gauge tagged by user ID +would trigger thousands of assessments. Assess the name once; the verdict covers +every meter registered under it. A `per-identity` option stays for the rare case +(tag-set-specific verdicts), default off. + +Cache TTL: with name-level static metadata, re-running inference after TTL expiry +produces no new information. Default TTL is effectively infinite for a JVM lifetime; +the property exists for long-running processes during development iterations, and +the docs should not pretend re-scoring discovers anything new. + +## 5. Core API (micrometer module) ``` -JevMetricAssessor // orchestrates: extract -> dedupe -> evaluate -> cache -JevMeterAssessment // record: Meter.Id coordinates, 4 answer values, timestamp -JevMeterFilter implements MeterFilter // the Micrometer entry point -JevAssessmentCache // bounded LRU + TTL, keyed by stable meter identity +JevMetricAssessor // orchestrates: extract name -> dedupe -> evaluate -> cache +JevMeterAssessment // record: meter name, 4 answer values, timestamp +JevMeterFilter implements MeterFilter // reduce-mode entry point +JevAssessmentCache // bounded, keyed by meter name JevProtectedMetrics // explicit allowlist, bypasses inference entirely +JevMetricReport // structured review artifact (the primary output) +JevMetricReviewListener // hook: fires per verdict + on report completion ``` Key decisions: -- **Identity = normalized `Meter.Id`.** Name + tag key/value pairs sorted, plus unit. - Stable string key (like jevmetrics' cache identity) so the same meter never triggers - a second inference call. -- **Jev call: all four questions in ONE evaluate** (per API reference; ~500-800 input - tokens, ~0.6-2.4s via gateway). Relevance and redundancy are informational; +- **Jev call: all four questions in ONE evaluate** (per API reference; ~500-800 + input tokens, ~0.6-2.4s via gateway). Relevance and redundancy are informational; `keep` noul drives policy; recommended `action` is exposed but never overrides the keep-score threshold. -- **Async by design.** The filter must never block meter registration on a network - call. On first sight of a new meter: admit it (annotate) or apply cached/default - policy, enqueue the assessment, background worker calls Jev, cache updated. - Subsequent batches/registrations use the cached decision. -- **annotate mode**: original meter always registered. Companion gauges +- **Async always.** Registration never blocks on a network call. First sight of a + new name: admit, enqueue, background worker calls Jev, listener + cache update. +- **Review mode (default)**: everything registered; companion gauges `jev.metric.relevance`, `jev.metric.redundancy`, `jev.metric.keep_probability` - registered under the original meter's tags plus `metric.name` and `jev.model` - tags, mirroring jevmetrics. `jev.metric.recommended_action` as a gauge with an - `action` tag (keep=1). Protected meters get no companions. + (tags: `metric.name`, `jev.model`) plus `jev.metric.recommended_action` (tag: + `action`). AND the report: at a configurable point (default: 60s after startup + or on `JevMetricReport.request()`), emit one WARN-group log with the flagged + meters, and write `jev-metric-report.json` (assessments, reasons, model id, + timestamp) for humans to read in a PR or code review. - **reduce mode**: `MeterFilter.reply(deny)` when cached keepProbability < - threshold. Before the first assessment lands, default is ADMIT (fail-open), so a - Jev outage can never blackhole telemetry. Protection allowlist always wins. -- **Failure policy**: any Jev error → log at WARN once per identity, meter stays - admitted, retry with exponential backoff. Never throw out of `accept()`. - -## 5. Spring Boot auto-config (starter module) + threshold. Documented limits, stated in the README, not buried: + (a) most meters register at startup before any verdict exists, so with fail-open + defaults the filter only bites on registrations AFTER the first assessment wave + (dynamic meters, second phase of a lazy-init app) or on the next boot with a + persistent cache; (b) fail-closed would make a Jev outage a telemetry blackhole + during incidents, so fail-open is mandatory, which caps reduce mode at + "gradual + advisory". An in-memory cache seed file (optional, off by default) + can carry verdicts across restarts for teams who want filtering from boot. +- **Failure policy**: any Jev error -> log WARN once per name, meter stays + admitted, exponential backoff. Never throw out of `accept()`. + +## 6. Spring Boot auto-config (starter module) ``` typesafe.jev.metrics.enabled (default true when a key exists) -typesafe.jev.metrics.mode annotate | reduce (default annotate) +typesafe.jev.metrics.mode review | reduce (default review; old name + "annotate" aliased) typesafe.jev.metrics.keep-threshold 0.0-1.0, default 0.5 (reduce mode only) -typesafe.jev.metrics.protected-names list of exact meter names, always admitted -typesafe.jev.metrics.cache-max LRU size, default 1000 -typesafe.jev.metrics.cache-ttl default 24h +typesafe.jev.metrics.protected-names exact meter names, always admitted +typesafe.jev.metrics.cache-max default 500 (name-level) +typesafe.jev.metrics.cache-ttl default: no expiry +typesafe.jev.metrics.report-delay default 60s; 0 disables scheduled report +typesafe.jev.metrics.report-file default none (log-only) +typesafe.jev.metrics.seed-file optional verdict cache across restarts typesafe.jev.metrics.worker-threads default 1 ``` Backs off with the same `TypeSafeApiKeyCondition` pattern as the existing client -bean (property OR `TYPESAFE_API_KEY` env), `@ConditionalOnMissingBean` on the filter -so user beans win. `ApplicationContextRunner` tests for both modes and for backoff. +bean (property OR `TYPESAFE_API_KEY` env), `@ConditionalOnMissingBean` on the +filter/listener beans so user beans win. `ApplicationContextRunner` tests for both +modes and backoff. -## 6. Testing +## 7. Testing -Mirror jevmetrics' test list, mapped to Micrometer, on the existing fake-transport -pattern (no key, no network): +Mirror the jevmetrics test list, mapped to Micrometer, on the existing +fake-transport pattern (no key, no network): - response validation: malformed/missing keep answer -> failure path, meter admitted -- mode normalization: unknown mode string -> annotate + WARN +- mode normalization: unknown/legacy mode string -> review + WARN - policy: threshold boundary (== threshold keeps), protected-name bypass -- payload preservation: counter/gauge/timer/long-task-timer all pass through - untouched in annotate mode; companion gauges appear with correct tags -- cache: identity dedupe (two `Meter.Id`s differing only in tag ORDER share one - entry), TTL expiry, eviction at max size -- queue saturation: unbounded burst of new meters -> bounded work queue, oldest - dropped, no OOM, no registration blocking +- payload preservation: counter/gauge/timer/long-task-timer untouched in review + mode; companion gauges appear once per NAME, not per tag set +- identity: two Meter.Ids same name different tags share one assessment +- cache: eviction at max, TTL only when explicitly configured +- queue saturation: burst of new names -> bounded queue, no OOM, no blocking - failure recovery: transport throws -> WARN once, backoff, later success caches +- report: JSON shape, flagged-list correctness vs raw answers, report-delay=0 off +- reduce limits (regression for honesty): startup-burst meters NOT denied on boot 1 + with empty cache; denied on boot 2 with seed file; Jev outage admits everything - auto-config: runner tests for both modes, backoff without key, user bean wins -- Surefire 3.2.5 pinned in the module pom; assert on "Tests run" counts, not exit code +- Surefire 3.2.5 pinned in the module pom; assert on "Tests run" counts Live smoke test env-gated (`TYPESAFE_API_KEY` + `assumeTrue`), same as core. -## 7. Milestones - -1. M1: micrometer module - identity, cache, assessor, filter, annotate mode, tests -2. M2: reduce mode + protection rules + policy tests -3. M3: starter auto-config + runner tests + README section -4. M4: examples app + "I built jevmetrics-style triage for Spring Boot" article arc - (gate through check_article_safety.py like every article) - -## 8. Open questions - -- Version line: this is a new module; propose shipping inside 0.3.0 (module addition - is minor, not patch). -- Optional second target later: an OpenTelemetry Java SDK extension for non-Micrometer - apps; explicitly out of scope for M1-M3. -- Naming: `jev.metric.*` companion gauge names copied from jevmetrics for ecosystem - recognizability; alternative is a `typesafe.metric.*` prefix. Default proposal: - keep `jev.metric.*` (the model is Jev, regardless of serving backend). +## 8. Milestones + +1. M1: micrometer module - name-level identity, cache, assessor, review mode, + report, tests +2. M2: reduce mode + protection rules + seed file + the honesty regression tests +3. M3: starter auto-config + runner tests + README (with the reduce-limits section + as a first-class section, not a footnote) +4. M4: examples app + article arc. Angle: "metric linting with a decision model", + not "cut your metrics bill". Gate through check_article_safety.py. + +## 9. Open questions + +- Version line: new module; propose shipping inside 0.3.0. +- Optional second target later: an OpenTelemetry Java SDK extension for + non-Micrometer apps, and/or an OTel Collector route if we ever want the + fleet-level vantage point jevmetrics has. Out of scope for M1-M3. +- Companion gauge naming: keep jevmetrics' `jev.metric.*` for recognizability + (recommended); alternative `typesafe.metric.*`. +- Report delivery: log + JSON file for M1; a build/CI plugin (fail the build on a + new "dead" meter) is a plausible M5 if the report proves useful in practice.