fix: keep sysmon responsive and harden runtime backtrace refresh - #295
Merged
Conversation
Refresh PID analyzers before compiling scripts for newly mapped libraries. Bound zero-sized symbols at the next address and preserve normalized zero-bias executable frames when coordinator lookup is unavailable.
Retain validated descriptors through parsing, including proc-root paths across mount namespaces, and isolate discovery so timeouts cannot block event handling or runtime shutdown. Bound retries for terminal observations and stop scheduling when the module limit is reached. Publish CFI to current trace actors, retain completed rows for later traces, and serialize initial and incremental writes to shared pinned maps. Publish runtime symbols with cache invalidation after CFI publication.
Bound ring-buffer draining so periodic reconciliation runs under sustained map-change load. Coalesce per-process map notifications and cache namespace PID scans for the noisy path while keeping lifecycle resolution fresh. Add a regression stressor that reproduces missing target offsets on the old implementation.
Move /proc scanning and offset refreshes to a dedicated worker so ring and perf collectors continue draining under load. Keep lifecycle work separate from coalesced map changes, and retain the first pending refresh deadline under continuous mmap traffic. Keep target-only offset updates from replacing complete process module ranges.
Publish immutable process-module snapshots for event rendering. Keep rendering off the coordinator lock and key caches by mapping generation. Require complete late-loaded-module symbolization in e2e coverage.
swananan
force-pushed
the
fix/runtime-backtrace-refresh-followups
branch
from
September 5, 2026 15:47
06ed101 to
1e39eb3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow up on #292, based on the current
main(1420f7f). The already-mergedruntime-loading commits are excluded from this PR; only the additional fixes
and sysmon improvements remain.
event collection from
/procscanning and offset refresh work.without waiting for the process coordinator lock.
and keep normalized non-PIE frames resolvable.
terminal-observation retry bounds, and the runtime-module limit.
serialize all writers to shared pinned CFI maps.
and publish runtime symbols at the renderer-cache invalidation boundary.
/proc/PID/rootmount-namespace access without reopening a same-named host library.
Validation
-D warningsand-D clippy::undocumented_unsafe_blocks.host -> docker-privatein runner job5bde05be7838. Coverage now checksfully symbolized blocks and the last ten shared-library backtraces.
cargo test --all-featurespasses (946 tests), with Rustcompatibility toolchains required and
dwarf-toolbuilt first.affected-crate strict Clippy and unit tests (174 and 73 tests respectively).
fd30db271c75and full host-to-private runnerjob
d83803938a23both pass all 32 backtrace tests, concurrent CFIpublication, and all late-start globals cases. Both then stop at
rust_script_execution: 33 passed and 16 failed. The failing test namesand assertion locations match pre-fix job
04d5a1bc8db7exactly. Laterlocal test binaries are not reached; CI runs the complete suite.
Earlier local full jobs
e7b4ca35a884and04d5a1bc8db7shared 16 Rustvalue-rendering failures. Those are tracked separately from the dlopen
backtrace CI failures fixed here.
All five commit messages are wrapped at 80 columns or fewer (maximum: 77).
Original author and committer timestamps are preserved.
CI outcome
All ten checks pass on
1e39eb3:Rustfmt, strict Clippy, Test Suite, Build, and Standard E2E all pass.
Standard E2E reports 342 passed and zero failed.
all three full topologies pass (342 tests each), along with host-PID smoke.
passes.
Both dlopen backtrace cases pass in Standard E2E and all three full container
topologies. The Rust value-rendering group also passes all 49 tests in each
CI topology, unlike the local runner environment noted above.
The first CI attempt exposed three concrete causes: symbol visibility
preceded renderer cache invalidation, periodic target-only updates discarded
other process ranges, and canonicalizing a proc-root path selected the host
libc. The amended commits fix those causes while preserving identity
validation, without increasing test timeouts or reducing event rates.