feat: implement the Telemetry API - #183
Conversation
Extensions can subscribe to the Telemetry API, but the emulator answers with Telemetry.NotSupported and never delivers anything, so extensions that consume telemetry cannot be exercised locally at all. This is issue aws#94. The pieces were nearly all present. NewServer already mounts the real TelemetryAPIRouter when EnableTelemetryAPI is set; StandaloneEventsAPI already builds records for the documented event types; StandaloneLogsEgressAPI already distinguishes a function's output from an extension's. What was missing was an implementation of telemetry.SubscriptionAPI -- only NoOpSubscriptionAPI exists -- and something to deliver events to subscribers. TelemetrySubscriptionService fills that gap. It validates subscriptions, honors the types filter and all three buffering limits, rewrites the sandbox hostname that only resolves inside a real execution environment, and posts batches to each subscriber. Events reach it through a dispatcher hook on the events API, so subscribers receive platform records as objects and log lines as strings, which is the distinction the API defines. Three behaviors worth calling out, each learned by comparing against telemetry captured from a real function: - The sandbox calls TurnOff() once initialization ends. That closes the subscription window; it must not stop delivery to extensions already subscribed, or everything after init is lost. - Initialization telemetry is emitted before an extension has had the chance to subscribe. Real Lambda delivers it regardless, so events produced before the first subscription are held, bounded, and replayed to each new subscriber. Without this, platform.initStart never arrives, which is exactly what an extension reporting cold starts needs. - Nothing called EventsAPI.SendReport, so platform.report was never produced even though the emulator prints a REPORT line. It is now reported where that line is printed. platform.end is deliberately not emitted: it does not appear in telemetry captured from a real function under the current schema version. Console output is unchanged. The logs egress tees to stdout as well as reporting telemetry, so docker logs still shows the runtime's and extensions' output. Verified by running an extension that records every delivered request body, inside the emulator, and diffing the result against telemetry captured from a real Lambda function invoked with the same handler: the set of event types now matches exactly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
![aws-sam-tooling-bot[bot]](https://avatars.githubusercontent.com/b/1290?s=60&v=4){kind=small}
| log.WithError(err).Warn("Telemetry API: could not encode a batch") | ||
| return | ||
| } | ||
| response, err := http.Post(sub.destination, "application/json", bytes.NewReader(body)) |
There was a problem hiding this comment.
[RESOURCE_MANAGEMENT] Delivery uses http.Post, which calls http.DefaultClient with a zero timeout — it will block indefinitely if a subscriber accepts the connection and never responds (or is slow). Because deliver is invoked synchronously from enqueue when maxItems/maxBytes is reached, and enqueue is called on the caller's goroutine of Dispatch (which runs on the event‑production path via StandaloneEventsAPI.appendEvent), a hung subscriber can stall the emulator's event pipeline. A misbehaving extension running under RIE can wedge the whole runtime.
Use a client with an explicit timeout:
var telemetryHTTPClient = &http.Client{Timeout: 10 time.Second}
// ...
response, err := telemetryHTTPClient.Post(sub.destination, "application/json", bytes.NewReader(body))There was a problem hiding this comment.
Fixed in 7f0923d. Delivery now uses a dedicated client with a timeout rather than http.DefaultClient.
I went with 5s rather than 10s: since a size-triggered batch is delivered on the goroutine producing the event, the timeout is how long the sandbox's event pipeline can be stalled, and for a local emulator giving up sooner seemed better than waiting longer. Happy to change it if you'd rather match something.
TestDeliveryToAHungSubscriberDoesNotBlockForever covers it with a subscriber that accepts the connection and never answers.
🤖 Generated with Claude Code
| telemetryEvents.Dispatch(standalonetelemetry.SandboxEvent{ | ||
| Time: now, | ||
| Type: "platform.report", | ||
| PlatformEvent: map[string]interface{}{ |
There was a problem hiding this comment.
[BUG] printEndReports emits platform.report with "status": "success" unconditionally, but the same function is called from the timeout path in InvokeHandler (case rapidcore.ErrInvokeTimeout: printEndReports(...)). A timed‑out invoke will be reported to subscribers as a successful one, which is exactly the signal extensions like ADOT/Datadog use to alarm on. Real Lambda emits "status": "timeout" (and "error" for failures).
Thread the status through the call, e.g.:
func printEndReports(invokeId, initDuration, memorySize string, invokeStart time.Time, timeoutDuration time.Duration, status string) {
// ...
"status": status,and pass "timeout" from the ErrInvokeTimeout branch and "success" from the normal completion path.
There was a problem hiding this comment.
yes, on timeout, the status should be "timeout"
There was a problem hiding this comment.
Fixed in 7f0923d, and thanks @luben for confirming. The status is threaded through printEndReports, with "timeout" from the ErrInvokeTimeout branch and "success" from the normal path. TestReportRecordCarriesTheInvocationStatus covers both.
🤖 Generated with Claude Code
| "requestId": invokeId, | ||
| "status": "success", | ||
| "metrics": map[string]interface{}{ | ||
| "durationMs": invokeDuration, |
There was a problem hiding this comment.
[BUG] memorySize is a string (from GetenvWithDefault("AWSLAMBDA_FUNCTION_MEMORY_SIZE", "3008")), so both memorySizeMB and maxMemoryUsedMB are serialized as JSON strings ("3008") rather than numbers. The Telemetry API schema defines these as numeric integers; consumers that decode them with typed structs or numeric JSON paths (ADOT, Datadog, X‑Ray forwarders, etc.) will fail or silently drop these metrics. Parse to an int before emitting:
memorySizeMB, := strconv.Atoi(memorySize)
// ...
"metrics": map[string]interface{}{
"durationMs": invokeDuration,
"billedDurationMs": math.Ceil(invokeDuration),
"memorySizeMB": memorySizeMB,
"maxMemoryUsedMB": memorySizeMB,
},There was a problem hiding this comment.
Fixed in 7f0923d. The memory size is parsed to an int, so the metrics serialize as numbers; TestReportRecordMetricsAreNumbers decodes the record into a typed struct, which is what would have failed before.
While confirming against telemetry captured from a real function I also noticed the report should carry initDurationMs on a cold start, so that's added, present only when the invocation initialized the environment.
Worth admitting this one found a hole in my own verification: I had compared which event types the emulator produced against the real capture, but not the field types within each record — which is exactly where this lived. I've since compared field by field, and the remaining differences are in the PR description.
🤖 Generated with Claude Code
| record.Record = event.LogMessage | ||
| } | ||
|
|
||
| if len(subscriptions) == 0 { |
There was a problem hiding this comment.
[CONCURRENCY] Dispatch snapshots the subscription map under the lock, then re-acquires the lock later to append to earlyEvents only if the snapshot was empty. A Subscribe call interleaving between those two lock sections drops the event:
Dispatch(e)takes the lock, seessubscriptionsis empty, releases.Subscribetakes the lock, inserts a new subscription, copiesearlyEvents(which does not yet containe), releases, replays copied events (nothing).Dispatchtakes the lock again and appendsetoearlyEvents— no live subscription will ever be handede, and furtherSubscribecalls may not occur.
Because subscribes happen during extension init and the sandbox produces platform events during that same window, this is exactly the ordering the code is meant to handle. Decide the empty-vs-non-empty branch under a single held lock:
s.lock.Lock()
if len(s.subscriptions) == 0 {
if len(s.earlyEvents) < maxEarlyEvents {
s.earlyEvents = append(s.earlyEvents, record)
}
s.lock.Unlock()
return
}
subscriptions := make([]subscription, 0, len(s.subscriptions))
for , sub := range s.subscriptions {
subscriptions = append(subscriptions, sub)
}
s.lock.Unlock()There was a problem hiding this comment.
Fixed in 7f0923d. The buffer-versus-deliver decision now happens under a single held lock, so a Subscribe can't interleave between the two sections and strand the event.
One caveat I'd rather state than gloss: I could not get a test to reproduce this. The two lock sections were adjacent with almost no work between them, and 400 attempts of concurrent Subscribe/Dispatch never hit the window. There is a concurrency test for the race detector, but it passes against the buggy version too, so it is not a regression test — this fix rests on reading the code, and on your analysis above.
🤖 Generated with Claude Code
Four problems, all reachable in normal use: Delivery used http.DefaultClient, which has no timeout. Batches that reach a size limit are delivered on the goroutine producing the event, so a subscriber that accepted a connection and never answered would stall the sandbox's event pipeline. Delivery now uses a client that gives up. platform.report was emitted with status success unconditionally, including from the invoke-timeout path, so a timed-out invocation was reported as a successful one -- the opposite of what an extension watching that field needs. The status is now threaded through. The report's memory metrics were serialized as JSON strings, because the memory size arrives as one. The Telemetry API defines them as numbers, and a consumer decoding into a typed struct rejects strings. They are parsed now, and the report also carries initDurationMs on a cold start, as a real function's does. Dispatch decided whether to buffer or deliver in one lock section and acted in another. A Subscribe interleaving between the two would insert itself, replay a copy of the buffer that did not yet hold the event, and leave the event buffered for nobody -- precisely the ordering the buffer exists to handle. The decision now happens under a single held lock. Tests cover the timeout, the status, the numeric metrics and the cold-start figure. The lost-event window is not covered: the two lock sections it needed to interleave between were adjacent and 400 attempts never hit it, so that fix rests on reading the code rather than on a failing test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
@vicheey , can you review it. From telemetry perspective it looks good to me, but I cannot asses if it fits well in the architecture. |
|
Thank you @lizthegrey for your contribution. I am reviewing the change internally with our team. There are a few logistically items that we have to address internally before we release the change. I'll keep you posted once we are ready. |
There was a problem hiding this comment.
Code Review Results
Reviewed: ddf2bc4..2c99593
Files: 8
Comments: 3
Comments on lines outside the diff:
[internal/lambda/rapidcore/standalone/telemetry/events_api.go:298] [RESOURCE_MANAGEMENT] appendEvent keeps appending to s.eventLog.Events with no bound, and the only drain, FetchTailLogs, is not called anywhere in this repository (grep finds the StandaloneEventsAPI implementation, the interop.EventsAPI interface entry, and the NoOpEventsAPI stub — no callers). That was harmless before this PR because RIE ran with &telemetry.NoOpEventsAPI{} and NoOpLogsEgressAPI, both of which discard everything.
After this PR, run.go installs StandaloneEventsAPI plus teeLogsEgressAPI, so every line of runtime and extension output now flows through sendLogEvent → appendEvent and is retained in that slice for the whole lifetime of the container process. A long-running RIE container with a chatty function grows this slice without limit, and it happens for all RIE users, including those who never subscribe an extension — earlyEvents is capped at maxEarlyEvents, but the event log behind it is not.
Since the dispatcher is now what actually delivers records, the log does not need to accumulate when one is attached:
func (s StandaloneEventsAPI) appendEvent(event SandboxEvent) {
s.lock.Lock()
dispatcher := s.dispatcher
if dispatcher == nil {
// Nothing drains eventLog when a dispatcher owns delivery.
s.eventLog.Events = append(s.eventLog.Events, event)
}
s.lock.Unlock()
if dispatcher != nil {
dispatcher.Dispatch(event)
}
}If the event log needs to stay populated for other consumers, bounding it (as earlyEvents is bounded) would work too — but it should not be unbounded for the life of the process.
| off := s.off | ||
| s.lock.Unlock() | ||
| if off { | ||
| return nil, http.StatusBadRequest, nil, fmt.Errorf("%s", s.GetServiceClosedErrorMessage()) |
There was a problem hiding this comment.
[BUG] The closed-window path returns an anonymous error, so the handler never recognises it as the "service off" case. runtimeLogsHandler.ServeHTTP compares the error by identity:
respBody, status, headers, err := h.telemetrySubscription.Subscribe(...)
if err != nil {
switch err {
case telemetry.ErrTelemetryServiceOff:
rendering.RenderForbiddenWithTypeMsg(writer, request,
h.telemetrySubscription.GetServiceClosedErrorType(), "%s", ...)
default:
rendering.RenderInternalServerError(writer, request)Because fmt.Errorf("%s", s.GetServiceClosedErrorMessage()) is never equal to telemetry.ErrTelemetryServiceOff, a subscription attempted after TurnOff() renders a generic 500 Internal Server Error and increments SubscribeServerErr. The http.StatusBadRequest returned here is discarded, and GetServiceClosedErrorType() / GetServiceClosedErrorMessage() — implemented a few lines below specifically for this — are never used. Real Lambda answers 403 with {"errorType":"Telemetry.SubscriptionClosed"}, which is what extension SDKs branch on to distinguish "you subscribed too late" from "the emulator broke".
The telemetry package is already imported in this file for the interface assertion, so:
if off {
return nil, http.StatusForbidden, nil, telemetry.ErrTelemetryServiceOff
}Note that TestTurnOffStopsSubscriptionsNotDelivery currently asserts http.StatusBadRequest, which locks in the wrong behaviour; it should assert errors.Is(err, telemetry.ErrTelemetryServiceOff) instead.
| if parsed.Scheme != "http" { | ||
| return "", fmt.Errorf("unsupported scheme %q", parsed.Scheme) | ||
| } | ||
| if parsed.Hostname() == "sandbox" { |
There was a problem hiding this comment.
[BUG] Only the bare sandbox hostname is rewritten, but the documented destination host for the Telemetry API is sandbox.localdomain — that is the form used in the AWS subscription examples and by real consumers (the Datadog extension subscribes with http://sandbox.localdomain:8124/lambda/logs, for instance). Inside a real execution environment both names resolve via /etc/hosts; inside an RIE container neither does unless rewritten, so resolveDestination returns the URI unchanged, every deliveryClient.Post fails DNS resolution, and the only symptom is a Telemetry API: could not deliver to ... warning. That reproduces the exact "nothing is ever delivered" outcome this PR sets out to fix, for the extensions most likely to be tested with it.
switch parsed.Hostname() {
case "sandbox", "sandbox.localdomain":
host := "localhost"
if port := parsed.Port(); port != "" {
host += ":" + port
}
parsed.Host = host
}Worth adding a TestSandboxHostIsRewritten case for http://sandbox.localdomain:3000 alongside the existing ones.
Do you want me to proceed with fixing the further reviewbot findings? |
## Which problem is this PR solving?
A function instrumented with OpenTelemetry can't use this extension
today. Sending OTel telemetry from Lambda currently means either running
a collector alongside the function, which the application connects to
over gRPC, or routing telemetry through CloudWatch.
Meanwhile this extension already has a cheaper path: read what the
function writes to stdout, translate it, ship it. It just only
understood Honeycomb's own JSON.
This teaches it OTLP/JSON, so instrumenting with OTel and pointing the
exporter at stdout is enough — no collector process and no socket for
the application to connect to.
## Short description of the changes
Sixteen commits, each independently reviewable, in four groups.
**The feature.** `otlpjson` recognizes an OTLP export request and hands
it to husky; the Telemetry API receiver expands one such record into an
event per span or log record, leaving everything else handled exactly as
before. A later commit adds the `otlp-stdout` envelope — see below for
why.
**Two fixes worth separating out.** A record of JSON `null`, or a
message with no record at all, reached libhoney's `Add` with a nil value
and panicked it, costing the rest of that batch; `main` panics on both
shapes, so this predates the branch. And an export request that
translated to zero spans was handled by no path at all — neither turned
into events nor logged — so it vanished silently.
**Build and docs.** `-s -w` strips the symbol table and DWARF from the
layer zip Lambda downloads at cold start (see the size table below); the
README documents which exporters actually work, which took a correction
after the first version named one that doesn't.
**Testing**, which is most of the commit count: payloads captured from a
real Lambda and replayed through the handler, the extension running
inside a real Lambda runtime, and CI running that on both published
architectures.
Traces and logs. Not metrics.
### Two line formats, because OTLP/JSON alone only reaches Java
This started as OTLP/JSON only, and that turned out to cover one
language. Checking what each SDK can actually emit:
- **Java** can, via `OTEL_TRACES_EXPORTER=experimental-otlp/stdout`
(1.43.0+). Experimental, as named.
- **Node's** `ConsoleSpanExporter` calls `console.dir(…, {depth: 3})` —
Node's inspect format, not JSON, multi-line, elided below depth 3, and
documented as subject to change at any time.
- **Python's** emits the SDK's own span shape via `to_json()`,
multi-line by design since
[#505](open-telemetry/opentelemetry-python#505).
Writing adapters for those console formats would mean parsing output
their own maintainers call unstable and diagnostic-only, so this instead
accepts the [`otlp-stdout`
exporters](https://github.com/dev7a/serverless-otlp-forwarder) that
Node, Python and Rust do have. They emit one JSON line wrapping a
compressed, base64-encoded export request. That takes coverage from one
language to four.
The envelope's declared `content-type` and `content-encoding` are passed
to husky rather than assumed, so protobuf or JSON, gzip or zstd or
uncompressed all work, and a change to the exporters' defaults won't
silently break parsing. The signal comes from the `endpoint` the payload
was addressed to, since a compressed body can't be inspected for it.
Two things worth noting: it costs **nothing** in binary size, because
the protobuf decoder was already linked for the JSON path; and since the
payload is compressed, it fits far more spans into a line before hitting
Lambda's truncation limit — the constraint most likely to bite in
practice.
These are community packages, not part of OpenTelemetry proper. If we'd
rather not build on a third-party envelope, the alternative is Java-only
until upstream ships stdout exporters, and that's a reasonable call to
make in review.
### Why husky rather than a hand-written mapping
`husky/otlp` is the same library Honeycomb's OTLP ingest uses. Reusing
it means field naming, resource-attribute flattening, sample rate,
timestamps and dataset routing are *by construction* identical to what
the same spans would produce through the OTLP endpoint — there's no
second mapping to drift out of sync. The alternative was ~350 lines
re-implementing `trace.parent_id`, `duration_ms`, `span.kind`,
`meta.annotation_type` and friends, and owning that indefinitely.
### Size: husky costs 2.6 MiB, stripping refunds 3.3 MiB
husky pulls in otel-proto and the protobuf runtime, so this was the
first thing measured. x86_64, `zip -9`:
| build | binary | layer zip |
| --- | --- | --- |
| main, as shipped | 16.04 MiB | 7.13 MiB |
| main, `-s -w` | 12.02 MiB | 3.81 MiB |
| this branch, no strip | 26.70 MiB | 12.13 MiB |
| this branch, as it will ship | 19.41 MiB | **6.41 MiB** |
Stating this plainly rather than letting the two commits net out to an
implied win: **husky costs +2.6 MiB zipped and +7.4 MiB on disk, and
every user pays it at cold start whether or not they emit OTLP.** The
shipped layer still shrinks 7.13 → 6.41 MiB (−10%), but that's the
stripping paying for it, and the strip commit stands on its own — it
would be worth taking even if this feature were rejected.
Worth knowing for anyone re-measuring: of husky's cost, the bulk is the
OTLP generated structs plus the protobuf runtime, which resists
dead-code elimination because it registers types reflectively. The
scary-looking transitive deps (gonum, grpc-gateway,
collector-contrib/sampling) *are* eliminated and cost ~0.5 MiB, and
adding the logs entrypoint on top of traces costs nothing measurable.
There is no smaller subset of `husky/otlp` to import — it's a single
package.
### Translated telemetry keeps the extension's marker
Translated OTLP carries `lambda_extension.type`, which the OTLP endpoint
would not add. That is deliberate rather than an oversight: annotating
telemetry with the component that handled it is what Refinery does on
the way through, and it is how a query tells a span that arrived via
this layer from one sent to Honeycomb directly. Documented in the README
and pinned by a test.
### Dataset routing changes for OTLP events only
Spans go to the dataset named by their `service.name`, as they would via
the OTLP endpoint. `LIBHONEY_DATASET` remains the destination for
classic keys and for every non-OTLP record. This is a deliberate
behavior difference from everything else the extension emits; the
alternative — collapsing all services into one configured dataset —
seemed worse than matching the endpoint.
### `LogMessage.Record` is now `json.RawMessage`
This is the largest mechanical part of the diff and the part most worth
a look. The record is kept as raw bytes instead of being decoded to
`interface{}` first, so the translator sees exactly what the function
wrote. Decoding first would round-trip nanosecond timestamps through a
float64 and quietly lose the low bits —
`TestOTLPNumericNanosecondsKeepFullPrecision` covers that case. The cost
is that existing test fixtures now build records as wire JSON rather
than Go maps. No pre-existing test changed its expectations, only its
fixture syntax.
## Testing
Three layers, weakest evidence first.
**Unit tests.** Detection across 12 cases including the near-misses
(`resourceSpans` nested rather than top level, libhoney envelopes,
non-JSON lines); translation of traces and logs; classic-versus-E&S
dataset routing, including husky's asymmetry between the two signals;
husky's sentinel errors rather than merely "an error occurred". Through
the HTTP handler: both of Lambda's log formats, multi-span fan-out,
platform records, malformed OTLP, export requests that translate to
nothing, and record shapes that used to panic libhoney.
**Replay of telemetry captured from a real Lambda function.** Every
fixture used to be one I wrote, which meant the tests confirmed my
beliefs about the wire format rather than the format itself — and one of
those beliefs was already wrong. `telemetryapi/testdata/capture/`
deploys a throwaway function that writes each shape to stdout, records
what the Telemetry API actually delivers, and tears itself down; the
recordings are replayed through the handler. Both log formats are
captured, and asserted to produce **identical** events, which is the
property that makes log format a non-issue for users.
Two things that capture settled, neither of which was knowable by
reading: under JSON log format an already-JSON line arrives
**verbatim**, with no platform keys merged in (extra keys would make
every OTLP payload fail to parse, since protojson rejects unknown
fields); and on a custom runtime a non-JSON line arrives as a bare
string under both formats, so the `{timestamp, level, message}` unwrap
path is still covered only by hand-written tests.
**The extension running inside a real Lambda runtime.** `make test-rie`
builds the extension into the Lambda base image at `/opt/extensions`,
and the platform starts it, registers it, and delivers telemetry over
the real Extensions and Telemetry APIs. The events it sends are decoded
and asserted on: OTLP/JSON traces and logs, the `otlp-stdout` envelope,
a libhoney envelope and a plain log line, plus the rule that only
translated telemetry routes away from the configured dataset. It also
covers the registration lifecycle, including the Lambda Managed
Instances path whose mishandling made an earlier release unusable.
This runs in CI on **both x86_64 and arm64** — arm64 layers are
published and nothing exercised them before — and releases now require
it. The emulator bundled in the base image stubs the Telemetry API, so
the suite builds one that implements it, pinned to a commit;
`EMULATOR_REPO`/`EMULATOR_REF` point it elsewhere, including at a local
checkout.
Claims were checked by mutation rather than assertion — breaking the
implementation and confirming a test notices. The nanosecond-precision
guard, the sample rate reaching the event, the null-record guard, the
function-type gate, the per-event dataset override, OTLP detection and
envelope support all fail their tests when reverted.
**What is still not tested.** The extension itself has never run on a
real Lambda: the capture above deployed a purpose-built recorder, not
this code, and an emulator is not the platform. The Java exporter
guidance is reasoned from upstream sources rather than observed in AWS.
Both remain reasons to try this on real functions before recommending it
to anyone.
### Two rounds of review already applied
An adversarial review pass found things worth recording, since they
shaped the diff:
- The first version of the README named Java's
`OtlpJsonLoggingSpanExporter`. That exporter emits lines with **no
`resourceSpans` wrapper**
([opentelemetry-java#6749](open-telemetry/opentelemetry-java#6749))
and writes through `java.util.logging`, so following the original docs
would have silently produced nothing. `experimental-otlp/stdout` is the
value that works.
- An export request translating to zero spans vanished entirely —
handled by no path, logged by nothing. Now warns and falls through.
- husky routes the signals asymmetrically for classic keys (spans honor
the configured dataset, log records still prefer `service.name`).
Documented and pinned by a test.
- Telling users `LIBHONEY_DATASET` wasn't their destination would have
led some to unset it, which disables the extension completely. The
README now says to keep it.
- A record of JSON `null` panics libhoney and costs the rest of the
batch. **This one predates the branch** — `main` panics identically, and
on the absent-`record` case too. Fixed here since the surrounding code
was already being touched.
## Open questions for review
- **Depending on community exporters.** The `otlp-stdout` packages that
give Node, Python and Rust a working path are from
serverless-otlp-forwarder, not OpenTelemetry proper. Accepting their
envelope is what takes this beyond Java. If we would rather only support
formats that come from upstream, this is Java-only until the OTLP File
exporters stabilize — a legitimate call to make.
- **Truncation stays silent.** Lambda truncates long log lines, and a
truncated payload arrives as a single event carrying the broken text in
a `record` field: no spans, no warning. The README says to keep batches
small, which is advice rather than a safeguard. Should the extension
recognize a truncated OTLP payload and say so?
- **The emulator suite points at a fork.** `test/rie` builds an emulator
with Telemetry API support from a personal fork, pinned to a commit,
because the one in the Lambda base image stubs that API. If [the
upstream
PR](aws/aws-lambda-runtime-interface-emulator#183)
lands, that constant becomes an upstream ref. Reviewers may reasonably
want this suite gated differently until then.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
3 participants
Implements the Telemetry API. Fixes #94.
Today an extension can call
PUT /2022-07-01/telemetryand gets202with{"errorType":"Telemetry.NotSupported"}, and nothing is ever delivered. Any extension whose purpose is consuming telemetry — ADOT, Datadog, Honeycomb's — can't be exercised locally at all.Approach
Nearly all of the machinery was already here:
NewServeralready mounts the realTelemetryAPIRouterwhenEnableTelemetryAPIis set, falling back to the stub otherwise.sandbox_builder.gohardcodes it tofalseand nothing callsSetTelemetrySubscription, which is what would flip it.StandaloneEventsAPIalready builds records for the documented event types —platform.initStart,platform.start,platform.runtimeDone,platform.extension, and so on.StandaloneLogsEgressAPIalready separates a function's output from an extension's own.What was missing was an implementation of
telemetry.SubscriptionAPI(onlyNoOpSubscriptionAPIexists publicly) and something to deliver events to subscribers. This adds both and wires them through the existing setters.TelemetrySubscriptionServicevalidates subscriptions, honors thetypesfilter and all three buffering limits (timeoutMs,maxItems,maxBytes), rewrites thesandboxhostname that only resolves inside a real execution environment, and POSTs batches to each subscriber. Events reach it via a dispatcher hook on the events API, so platform records arrive as objects and log lines as strings — the distinction the API defines and that consumers branch on.Console output is unchanged: the logs egress tees to stdout as well as reporting telemetry, so
docker logsstill shows everything it did before.Three things that only showed up against real telemetry
I captured telemetry from an actual deployed Lambda function and diffed it against what the emulator produced. Each of these was a bug found that way:
TurnOff()closes the subscription window, it does not stop delivery. The sandbox calls it once initialization finishes (handlers.go:397, "no more agents can be subscribed"). My first version treated it as a delivery kill switch, so everything after init silently vanished.platform.initStartnever arrives — precisely what an extension measuring cold starts needs.EventsAPI.SendReport, soplatform.reportwas never produced despite the emulator printing aREPORTline. It's now reported where that line is printed.platform.endis deliberately not emitted: it doesn't appear in telemetry captured from a real function under the current schema version.Verification
An extension that records every delivered request body, run inside the emulator, diffed against telemetry captured from a real Lambda function invoked with the same handler. The set of delivered event types now matches exactly:
Covering
function,platform.initStart,platform.initRuntimeDone,platform.initReport,platform.start,platform.runtimeDone,platform.report,platform.extension, andplatform.telemetrySubscription.Comparing the records field by field rather than only their types — which is how the string-versus-number metrics slipped past my first pass — these differences remain:
platform.reportspansarray (extensionOverhead,responseLatency); the emulator has no such measurementsplatform.initReportstatus;interop.InitReportDatacarries no status, so the emulator does not have itplatform.startfunctionArnplatform.initStartfunctionArnandruntimeArnplatform.runtimeDoneinternalMetricsandtenantIdThe last three come from the existing record builders in
StandaloneEventsAPIrather than from this change, so I have left them alone rather than widen the diff — happy to follow up if you would rather they matched.Then end to end with Honeycomb's Lambda extension, which translates OTLP found in function stdout: it received the telemetry, translated the spans, and delivered them to a stand-in API with the correct per-service routing — the whole path exercised locally with no AWS involved.
Unit tests cover record shapes, type filtering, pre-subscription replay, the
TurnOffsemantics above,Clearon reset,maxItemsflushing, rejection of unusable subscription requests, and hostname rewriting.go test ./internal/lambda/...passes unchanged.Why not build on #137
@AndrewChubatiuk got there first in #137 and deserves the credit for pushing on this; I tried that branch before writing anything. I didn't extend it because the mechanism seemed hard to make faithful, and I'd rather explain that than quietly duplicate the work:
EnableTelemetryAPIstaysfalseand the response remains202 Telemetry.NotSupportedeven when telemetry will be delivered. An extension that checks the subscribe status treats that as failure — mine exited, and the emulator reportedExtension.Crash→Init failed.os.Stdout/os.Stderrprocess-wide and re-emitting every line, so all telemetry is typedfunction.START RequestId: ...arrives as a function log line where real Lambda sends a structuredplatform.start, and noplatform.*events are delivered at all.functiontelemetry. For an extension that logs while handling telemetry that's a feedback loop; I had to write my test captures to a file rather than stdout to measure anything.Concretely, same handler, same invoke: #137 delivers 10 messages all typed
function; real Lambda delivers 20 across nine types.None of that is a knock on the contribution — it unblocked the case it was tested against. It's that going through the existing
EventsAPI/LogsEgressAPIabstractions gets the types, the record shapes, and the subscribe response right by construction, rather than needing to reconstruct them from a line of text.@valerena, you mentioned discussing this with the internal Lambda team. If there's an internal implementation this should converge with, I'm happy to adapt or drop this in favour of it — the captured-from-real-Lambda comparison above is probably reusable either way as a conformance check.
Disclosure: I work at Honeycomb and maintain the extension linked above, so local Telemetry API support is directly useful to me. The captured-from-real-Lambda comparison is vendor-neutral, though, and I'd expect it to be just as useful for ADOT or any other telemetry-consuming extension.
🤖 Generated with Claude Code