[Investigation] Integrate FastTiming with standardized tracing and metrics

[simonrozsival]

Summary

FastTiming is a custom native timing system used by the MonoVM and CoreCLR Android hosts. It measures startup work such as runtime initialization, assembly loading and decompression, typemap lookup, registration, and native-to-managed initialization. Depending on debug.mono.log, it either writes events to logcat immediately or buffers them for a later mono.android.app.DUMP_TIMING_DATA broadcast.

We should investigate whether these measurements should instead use, or integrate with, standardized diagnostics and observability mechanisms such as:

• EventPipe/EventSource and .nettrace
• Activity/OpenTelemetry traces
• Meter/OpenTelemetry metrics
• Android Perfetto/ATrace where system-level startup correlation is valuable

This issue is intended to produce a design decision, not to assume that deleting FastTiming is necessarily the right result.

Why investigate this?

The current facility provides useful measurements, but it has accumulated implementation and usability debt:

• It uses a private event model and logcat text format rather than standard .NET diagnostics.
• timing=fast-bare requires the DumpTimingData broadcast receiver. The manifest overlay is gated by the internal _AndroidFastTiming MSBuild property, which has no default assignment or documented public equivalent.
• The debug.mono.timing duration= option is parsed and stored but never used.
• There are no focused tests for the recorder, event schema, output modes, or broadcast path.
• NativeAOT does not initialize or use FastTiming; the Android.Runtime.TimingLogger native entry points are unreachable stubs there.
• The current dynamically growing event buffer has a serious concurrency/correctness problem tracked by [CoreCLR] FastTiming::start_event corrupts the heap when debug.mono.log=timing (Scudo "invalid chunk state when deallocating") #12437.
• Plain timing mode duplicates runtime JIT diagnostics and has its own output reliability issue tracked by Generating methods.txt appears to stop writing to disk and is truncated #9693.

Standardized output could make the data available in existing tools and potentially in OpenTelemetry/Aspire, while reducing custom infrastructure. However, several important startup and cross-runtime constraints mean this is not a straightforward replacement.

Current timing coverage

FastTiming is implemented in:

• timing-internal.hh
• timing-internal.cc

The CoreCLR host initializes it before calling coreclr_initialize() and explicitly times that call:

• Host::init()

MonoVM has corresponding instrumentation in:

• monodroid-glue.cc

This gives FastTiming visibility into work that happens before the managed runtime and its diagnostics infrastructure are initialized, including Android environment setup, APK/assembly-store discovery, decompression, and portions of native runtime initialization.

Important distinction: semantic model vs. transport

Before selecting an API, we should distinguish what the data represents from how it is collected:

• Startup phases and nested operations are naturally trace spans.
• Stable aggregate durations can additionally be represented as histogram metrics.
• EventPipe is primarily an in-process event transport and collection mechanism.
• OpenTelemetry defines trace and metric models and exporters such as OTLP.
• Aspire consumes OTLP telemetry; it does not consume .nettrace files directly.
• Perfetto provides an Android system timeline but does not directly provide an OTLP path.

Metrics alone would lose ordering, nesting, and causal context. Assembly or type names would also be problematic metric dimensions because of cardinality. A likely model is detailed spans plus a small set of low-cardinality aggregate metrics.

EventPipe findings

EventPipe is available earlier than "after coreclr_initialize()":

1. coreclr_initialize() enters CoreCLR startup.
2. EEStartupHelper() calls EventPipeAdapter::Initialize().
3. The diagnostic server can pause for a startup collector.
4. GC, JIT, CoreLib, and other runtime initialization continue.
5. EventPipeAdapter::FinishInitialize() completes initialization.

Relevant runtime source:

• coreclr_initialize()
• EventPipeAdapter::Initialize() during EE startup
• EventPipeAdapter::FinishInitialize()
• ep_init() and environment-configured startup sessions

An EventPipe startup session can therefore capture runtime-internal JIT, GC, loader, and other events emitted during coreclr_initialize(). .NET for Android already documents startup collection through dotnet-trace and dotnet-dsrouter in Documentation/guides/tracing.md.

EventPipe does not currently replace all of FastTiming, however:

• It is not initialized before entering coreclr_initialize(), so it cannot directly record the earliest Android-host work.
• libcoreclr.so does not expose a supported native provider/write API through coreclrhost.h. Runtime-internal providers can write during startup, but the external Android native host cannot register and write a custom provider through a stable public ABI.
• EventPipe assigns an event's timestamp when it is written. Pre-runtime records flushed later could include their original timestamps/durations as payload, but their EventPipe envelope timestamps would represent the later flush.
• Existing runtime providers overlap JIT, GC, loader, and managed assembly activity, but not Android-specific work such as assembly-store decompression, typemap lookup, and JNI initialization.
• MonoVM supports EventPipe through its diagnostics component, but requiring diagnostics can increase packaged size and collection still has the same pre-managed/native-host gap.
• NativeAOT support and startup configuration need separate investigation; its Android host currently has neither FastTiming coverage nor equivalent wiring.

OpenTelemetry and metrics findings

ActivitySource/Activity is the closest standard semantic model for the ordered startup phases. Once managed diagnostics have been configured, buffered native records could be materialized as activities with explicit historical start/end times. This needs a prototype because:

• Native measurements use CLOCK_MONOTONIC_RAW, while Activity timestamps are wall-clock DateTimeOffset values. The clock mapping and its error must be characterized.
• Activities are only created when a listener is active. Native records would need to remain buffered until the application or platform configures the listener/OpenTelemetry SDK.
• Initializing an SDK/exporter during startup adds work to the startup being measured.
• Android apps do not automatically send telemetry to an Aspire dashboard. The application needs explicit OpenTelemetry/OTLP configuration and connectivity to the collector.
• Detailed assembly/type information raises cardinality, size, and potential metadata-disclosure concerns for production exporters.

Meter histograms could complement traces with a small stable set of measurements such as total runtime initialization, assembly loading, decompression, and typemap time. They are less suitable as the sole representation because a startup is a one-shot ordered sequence rather than only a latency distribution.

EventPipe and OTLP should not be treated as interchangeable. A managed EventSource can provide standard .nettrace events, while Activity and Meter provide OpenTelemetry-friendly semantics. If both outputs are required, we need either two emitters over one shared record model or a deliberate bridge.

Options to evaluate

A. Stabilize the existing implementation

Fix the current correctness issue, activation path, dead options, tests, and NativeAOT behavior while retaining logcat/file output.

Advantages

• Preserves a small native mechanism that works before managed initialization.
• Keeps the simple adb setprop and logcat workflow.
• Does not require diagnostics components, an external collector, or application OTel configuration.

Disadvantages

• Continues maintaining a private schema, storage implementation, and parser.
• Does not integrate naturally with .nettrace, OTLP, or Aspire.

B. Add an EventPipe/EventSource path

Use existing runtime events where they already describe the operation. Add Android-specific standardized events for the remaining phases, potentially flushing pre-runtime records after managed initialization.

Advantages

• Integrates with dotnet-trace, PerfView, and existing runtime traces.
• Avoids duplicating JIT/GC/loader instrumentation.
• MonoVM and CoreCLR already support the diagnostic protocol.

Disadvantages and blockers

• There is no supported external native EventPipe provider API today.
• Pre-runtime event timestamps cannot be represented as native EventPipe envelope timestamps when flushed later.
• Collection requires diagnostics support and, for interactive Android collection, usually dotnet-dsrouter.
• Cross-runtime and NativeAOT behavior must be designed explicitly.

C. Add OpenTelemetry activities and metrics

Keep a small native recorder for early events, then expose them as historical Activity spans and selected Meter histograms after managed telemetry is configured.

Advantages

• Uses standard tracing and metric models.
• Can integrate with OTLP backends and Aspire.
• Activities preserve parent/child phase structure; metrics support regression dashboards.

Disadvantages and blockers

• Requires application opt-in and an initialized listener/exporter.
• Adds exporter/SDK overhead and networking considerations.
• Requires monotonic-to-wall-clock timestamp mapping.
• Needs filtering, sampling, and metadata/cardinality rules.
• Does not automatically produce .nettrace with equivalent fidelity.

D. Add Android Perfetto/ATrace integration

Emit native startup slices into the Android system trace, possibly in addition to EventPipe or OpenTelemetry output.

Advantages

• Covers the pre-managed phase directly.
• Correlates .NET startup with Android process, scheduling, graphics, and system activity.

Disadvantages

• Uses a different collection workflow.
• Does not directly integrate with OTLP/Aspire.
• Still needs managed correlation and a cross-runtime event schema.

E. Hybrid shared recorder with pluggable outputs

Retain a bounded native record buffer as the source of truth for the earliest phases, but replace custom instrumentation/output duplication with one stable event schema and opt-in sinks:

• logcat compatibility output
• EventSource/EventPipe output
• Activity/Meter output
• optionally Perfetto slices

This may preserve current low-level coverage while allowing standardized tooling, but it also risks retaining too much complexity unless the sinks and ownership are carefully limited.

Questions the investigation should answer

1. Which existing runtime events duplicate current FastTiming events, and which Android-specific events remain necessary?
2. Is adding a supported native EventPipe provider/write bridge appropriate, or should native records always cross into managed code first?
3. Can pre-runtime timestamps be correlated accurately enough with EventPipe and Activity timelines?
4. Should startup details be modeled as EventSource events, Activity spans, metrics, Perfetto slices, or more than one output over a shared schema?
5. What is the disabled and enabled overhead of each option, including SDK/exporter initialization?
6. Can one design work across MonoVM, CoreCLR, and NativeAOT without requiring heavyweight diagnostics in normal applications?
7. What should remain available with only adb and logcat, without rebuilding an app or running a collector?
8. Which event details are safe and useful for production telemetry, and which should remain developer-only?
9. Can the broadcast receiver and _AndroidFastTiming property be removed, or must they be promoted to supported public behavior?
10. Should Android.Runtime.TimingLogger be retained, redirected to a standard API, deprecated, or made functional on NativeAOT?

Suggested investigation/prototype

1. Define a runtime-neutral schema for the existing startup phases.
2. Produce a coverage matrix for MonoVM, CoreCLR, and NativeAOT showing:
   • pre-runtime native events
   • runtime-internal events already available through EventPipe
   • post-runtime Android-specific events
3. Prototype a bounded native buffer that can flush one startup trace through:
   • EventSource/EventPipe, and
   • Activity/OpenTelemetry
4. Compare it with the current buffered timing=fast-bare output on real devices:
   • timestamp accuracy and ordering
   • missing/duplicated events
   • startup overhead when disabled and enabled
   • binary size and required diagnostics components
   • collection ergonomics
5. Decide whether to stabilize, integrate, deprecate, or replace FastTiming.

Acceptance criteria

• A documented decision identifies the target semantic model and collection transport(s).
• Coverage and limitations are documented for MonoVM, CoreCLR, and NativeAOT.
• Pre-coreclr_initialize() and equivalent pre-managed phases remain measurable.
• Existing EventPipe events are reused rather than duplicated where practical.
• Disabled/enabled startup overhead and timestamp accuracy are measured on devices.
• The developer-only versus production telemetry boundary is defined.
• A migration/compatibility plan exists for timing, timing=bare, timing=fast-bare, Android.Runtime.TimingLogger, and current log parsers.
• _AndroidFastTiming, the dump receiver, and duration= are either supported and tested or removed.
• [CoreCLR] FastTiming::start_event corrupts the heap when debug.mono.log=timing (Scudo "invalid chunk state when deallocating") #12437 is addressed independently of the longer-term design.

Related issues

• [CoreCLR] FastTiming::start_event corrupts the heap when debug.mono.log=timing (Scudo "invalid chunk state when deallocating") #12437
• [net11.0-android/CoreCLR] DOTNET_EnableEventPipe via AndroidEnvironment aborts app at startup (SIGABRT) instead of writing a trace #12161
• [NativeAOT] env var timing at startup #10705
• Generating methods.txt appears to stop writing to disk and is truncated #9693