ci(memtrack): benchmark memtrack's own tracking overhead - #512
Draft
not-matthias wants to merge 5 commits into
Draft
ci(memtrack): benchmark memtrack's own tracking overhead#512not-matthias wants to merge 5 commits into
not-matthias wants to merge 5 commits into
Conversation
`codspeed-memtrack track` pays a fixed cost per invocation (BPF program load plus uprobe/uretprobe attaches) on top of the tracked command, and nothing measured it so far, so wall-clock regressions in that overhead went unnoticed. Add a walltime config with three exec targets covering distinct workloads (read-only, allocation-heavy, I/O-heavy) and a CI job that runs them with the CLI and memtrack built from source.
Merging this PR will not alter performance
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| 🆕 | WallTime | memtrack track dd |
N/A | 1.4 s | N/A |
| 🆕 | WallTime | memtrack track ls |
N/A | 1.4 s | N/A |
| 🆕 | WallTime | memtrack track tar |
N/A | 5.2 s | N/A |
Comparing memtrack-walltime-benchmarks (8201908) with main (1dcc738)
The listing and dd's stderr were captured into the runner log once per round, which made the uploaded log 4.2 MB of noise. The tracked command string is run through `bash -c`, so a redirect inside it works.
Allocator entry points share addresses through aliases: `free`, `cfree` and `__libc_free` are one symbol in glibc, and the standard-probe sweep attaches all of the names it finds. Attaching `uprobe_free` twice at one address does not double the trap, since the kernel keeps a single uprobe per address with a list of consumers, but it does run the program twice per call and emit a duplicate free event: 607k events for 200k malloc/free pairs, 406k after this change. Measured on a malloc/free latency harness (p50 per pair, glibc): 1272 ns to 1162 ns, and one fewer link to attach and detach per aliased symbol.
The uprobe/uretprobe argument hand-off kept a hash map keyed by tid for every instrumented function, costing an update on entry and a lookup plus delete on return, and every hook re-resolved is_tracked() through further hashed lookups of the pid and its ancestors. Both now live in task-local storage, reached by a pointer chase off the task_struct instead of a hashed, bucket-locked lookup. One slot per entry point rather than a single shared one, since allocators call each other (glibc realloc reaches malloc) and nested calls on a thread must not clobber each other's saved arguments. A `valid` bitmask keeps a zero argument distinguishable from an absent one, so a return probe firing without its entry probe is still ignored. The tracked flag is only memoized when positive: pids are added to tracked_pids and never removed, so a tracked task stays tracked, while an untracked one may be registered later and must keep re-resolving. Measured on a malloc/free latency harness (p50 per pair, glibc): 2204 ns to 2064 ns.
Every submit called bpf_ringbuf_query(BPF_RB_AVAIL_DATA) to decide whether to force a consumer wakeup. That reads the consumer position, a cache line the polling thread on another CPU writes continuously, so each event paid a cross-CPU miss for a decision that only changes once per watermark. Count submitted bytes per CPU instead and force a wakeup whenever the watermark is crossed. Events are fixed size, so this is the same cadence the query approximated, decided entirely on the producer side with no shared cache line involved. A missing counter forces the wakeup rather than risking a stalled consumer. Measured on a malloc/free latency harness (p50 per pair, glibc): 2064 ns to 1102 ns, the largest of the three hot-path wins. Verified at 10M malloc/free pairs (20,006,217 events, ~800 MB through the 256 MB ring buffer) with the dropped-event counter still at zero, so batched wakeups keep up with a sustained high event rate.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
codspeed-memtrack trackpays a fixed cost per invocation — BPF program load plus uprobe/uretprobe attaches — on top of running the tracked command. Nothing measured that overhead, so wall-clock regressions in it went unnoticed.What
crates/memtrack/codspeed.yml: walltime benchmarks drivingcodspeed-memtrack trackover three workloads:ls -la /usr/lib/x86_64-linux-gnu— read-only, low allocationtar -cf ... /usr/lib/x86_64-linux-gnu— allocation- and I/O-heavy (many small reads)dd if=/dev/zero of=... bs=1M count=64— I/O-heavy, minimal allocation.github/workflows/ci.yml:memtrack-benchmarksjob that builds both the CLI and memtrack from source, grants memtrack the file capabilities (same sequence as thetestsjob), and runs the config withcodspeed run -m walltime. Added tocheck'sneeds.warmup-time: 5s/max-time: 60sare deliberately generous: the per-invocation attach cost alone is ~1.1s, so the walltime defaults tuned for near-instant commands would yield no usable rounds.Verification
Ran locally on Linux (NixOS, so the exec paths were substituted for local equivalents; everything else unmodified):
Measured medians over 3 rounds:
ls1.15s,tar1.96s,dd1.30s.