Runner vs OTel Overhead Findings (2026-05)¶

Status: event-rate boundary-finding update. This document summarizes the overhead follow-up evidence collected so far. It does not commit the generated measurement artifacts. Direct arm deltas are reported only for the matching linux-aarch64-6.8.0-117-generic host class.

Evidence Anchors¶

Generated artifacts from those runs were inspected as review artifacts only. They are intentionally not committed as benchmark evidence in this slice.

The failed diagnostic repeat is listed to explain the workflow-DX fix, not to replace the successful Slice 7 findings. The follow-up repeat passed with artifacts uploaded and the new artifact-success gate active. Together, the repeats show that the first-sample cgroup failure is not deterministic, while wall-clock decomposition still needs phase timing before it can support an additive claim.

The phase-timing runs are listed as diagnostics, not as replacement baselines. They validate the Slice 8 instrumentation and localize part of the Arm A / Arm C median gap, but Arm A again showed an unhealthy wall-clock tail in that dispatch.

The paired Slice 9 run is also diagnostic. It keeps Arm A and Arm C adjacent in one delegated job to reduce inter-dispatch drift. It does not replace the same-host baselines below, and its unhealthy tails keep wall-clock publication caveats in force.

The Slice 10 smoke runs are workflow and metadata validation only. They show that the event-rate sweep knobs reach the real delegated workload: both arms captured event-rate-sweep/worker-* kernel events, Arm C emitted assay.sweep.* trace metadata when span/event pressure was requested, and Arm A correctly recorded span_event_rate=baseline with target_span_events=0. They are too small to support slope, threshold, or benchmark findings.

The Slice 11 starter matrix is the first event-rate sweep findings slice. It is still small (n=5 adjacent pairs per cell), so it reports health, calibration, and threshold signals only. It does not publish a product benchmark or a new additive wall-clock decomposition.

The Slice 12 boundary-finding sweep is the widened event-rate pass. It ran the predeclared cells sequentially because the workflow concurrency group keeps only one active delegated overhead run at a time. All six completed runs had 5/5 measured samples per arm, 0 discarded samples, clean Runner health gates, and successful warm-up samples. The finding is a fidelity/health boundary result, not a benchmark result.

Same-Host Baselines¶

All three arms now have clean measurements on the same delegated assay-bpf-runner host class. The runs were dispatched separately, so they characterize a shared host class, not co-temporal variance.

The one-byte artifact-size spread between Slice 2 and Slice 3 is expected for freshly generated archives and traces. The useful claim is that the footprint is tiny and stable at this workload scale, not that archive bytes are deterministic across runs.

Same-Host Delta¶

Because Arm B and Arm C emitted the same host_class, a narrow same-host delta is now valid for this deterministic workload. The runs were not co-temporal, so this is still a host-class baseline comparison, not a product benchmark.

The wall-clock delta is the cost of the current dual-capture path on this host class: Runner archive capture plus the existing OTel trace around the deterministic workload. The Arm A section below records the runner-only decomposition attempt, but that decomposition is not stable enough to turn this delta into an additive wall-clock cost model.

Runner-Only Decomposition¶

Arm A adds the runner-only comparison point: assay runner-spike with Linux/eBPF archive capture and the deterministic OpenAI Agents fixture, but without OTel trace export. All three arms emitted the same host_class, but the runs were separate dispatches and Arm A uses the fixture-agent path rather than the OTel workload wrapper. Treat this as an experiment-scoped decomposition aid, not as a general additive cost model.

The decomposition read is:

• RSS: useful and stable. Arm A and Arm C differ by only 8,192 bytes at the median RSS level (0.007%). The observed same-host RSS increase over Arm B is therefore dominated by Runner capture rather than by adding the OTel trace wrapper around Runner capture.
• Wall-clock: inconclusive as an additive decomposition. The healthy Arm A repeat is still 121.683 ms slower at the median than Arm C, even though Arm A omits OTel trace export. That is not a meaningful "OTel adds negative overhead" result; it means the runner-only fixture path and dual-capture workload path need phase timing before the current data can be decomposed into additive cost buckets.

Phase-Timing Read¶

Slice 8 added experiment-scoped phase diagnostics via assay.experiment.runner_phase_timing.v0. Both Arm A and Arm C phase runs used the same linux-aarch64-6.8.0-117-generic host class and produced 20 valid samples with 0 discarded samples.

Median phase breakdown:

The summed phase medians explain about 34.540 ms of the 107.251 ms Arm A median wall-clock gap. The largest instrumented contributor is monitor_attach_ms (+38.284 ms for Arm A), but most of the median gap remains outside the current phase buckets as measured by wall median minus summed phase medians (+72.711 ms residual).

That means Slice 8 supports a narrower conclusion than an additive wall-clock decomposition: the Runner-internal phases do not fully explain why the runner-only Arm A path is slower than Arm C at the median. The wall-clock split remains unsuitable for a "Runner archive only + OTel trace export" additive claim.

Paired Residual Read¶

Slice 9 dispatched arm=paired-a-c in run 26479319306. The harness ran 20 adjacent counterbalanced pairs in one delegated job: odd pairs used Arm A then Arm C, even pairs used Arm C then Arm A. Both arms produced 20 valid samples, 0 discarded samples, ringbuf_drops=0, kernel_layer=complete, and cgroup_correlation=clean.

Median phase breakdown from the paired run:

This paired result changes the wall-clock read: the Slice 8 Arm A slower-than-Arm-C median gap does not reproduce when the arms run as adjacent counterbalanced pairs. In the paired run, Arm A is faster at the median and the per-sample residual medians differ by only 22.476 ms. The result points to inter-dispatch drift and measurement variance as material contributors to the earlier wall-clock anomaly.

The paired run does not justify a new additive wall-clock model: both paired tails are unhealthy (p99/median > 2.0), and the paired wall deltas have a wide spread. It does justify a stopping rule for this arc: wall-clock decomposition is not stable enough at n=20 on this runner to publish as an additive split. RSS remains the clean decomposition signal.

Event-Rate Starter Matrix¶

Slice 11 ran the predeclared five-cell paired A/C starter matrix. Every cell used repetitions=5, measure_rss=false, build_ebpf=true, and paired adjacent A/C order on the same linux-aarch64-6.8.0-117-generic host class. All cells passed the health gates: 5/5 valid samples per arm, 0 discarded samples, ringbuf_drops=0, kernel_layer=complete, and cgroup_correlation=clean.

The starter matrix is useful mainly because it did not find a failure boundary at these levels. Even the corner cell stayed clean: no ring-buffer drops, no degraded kernel layer, no cgroup-correlation failure, and a healthy tail ratio. The corner trace size grew to a median 6,636,746 bytes, while the Arm C archive remained small (3,905 bytes compressed median, 93,987 bytes extracted median).

The wall-clock medians still should not be read as a product benchmark. At n=5, paired medians remain sensitive to run-window variance. The publishable Slice 11 result is narrower:

• the sweep knobs calibrate correctly on real delegated infrastructure;
• high=100 kernel events produces the expected 100 worker files per sample in both Arm A and Arm C;
• high=100 span events produces exactly 100 assay.sweep.span_event entries per Arm C trace;
• no starter cell hit the health or tail failure boundary;
• larger OTel span payloads scale trace size clearly, but did not destabilize Runner health at this matrix budget.

Event-Rate Boundary Sweep¶

Slice 12 ran the predeclared widened A/C matrix with repetitions=5, warmup_iterations=1, measure_rss=false, build_ebpf=true, and timeout_seconds=300. Warm-up samples are review artifacts only; all measured cells below had 5 valid samples per arm and 0 discarded samples.

The kernel-capture branch stayed healthy through the widened cells: x1000 kernel worker files calibrated exactly for both arms, and the kc1000 cell stayed clean even at concurrency 16. The two kernel-only cells put Arm A's tail ratio in the warning band, but no cell crossed the fail boundary (p99/median > 2.0), and no Runner health gate degraded.

The span/event branch hit a fidelity boundary before timing can be interpreted. At both x500 and x1000, the Arm C trace retained exactly 128 assay.sweep.span_event entries per sample. That matches the OpenTelemetry JS SDK default span event count limit in this workload configuration, so the apparent wall-clock and trace-size behavior above 100 span events is not evidence that the system handled 500 or 1000 trace records. It is evidence that the default OTel span limit preserves only 128 events.

Mechanism verification:

• The OpenTelemetry SDK environment-variable specification defines OTEL_SPAN_EVENT_COUNT_LIMIT as the maximum span-event count, with default 128: https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#span-limits.
• The OpenTelemetry Trace SDK specification likewise defines EventCountLimit (Default=128) under Span Limits: https://opentelemetry.io/docs/specs/otel/trace/sdk/#span-limits.
• The checked-in workload declares @opentelemetry/sdk-trace-base@^2.0.0 in workload/package.json, and the v1 findings record that dependency as resolved to 2.7.x at install time. Its BasicTracerProvider setup does not pass a custom spanLimits override, so default SDK limits apply unless the environment overrides them.
• The retained event indexes prove the cap is on span events, not on the Runner archive or JSON writer. For s500, every Arm C measured trace retained indexes 372..499; for s1000 and corner-lite, every Arm C measured trace retained indexes 872..999. That is the last 128 events, matching the OTel JS behavior of dropping the oldest event once eventCountLimit is reached. This last-N pattern is consistent with FIFO truncation under Span Limits, not with random sampling, first-N truncation, or JSON serialization loss.
• A local workload repro confirmed the mechanism: with OTEL_SPAN_EVENT_COUNT_LIMIT unset, --sweep-span-events 500 retained 128 events (372..499); with OTEL_SPAN_EVENT_COUNT_LIMIT=1000, the same workload retained all 500 events (0..499). With target 1000 and limit 1000, it retained all 1000 events (0..999).

The corner-lite cell is therefore a mixed result: Runner kernel capture stayed healthy at 1000 worker files with large payloads and concurrency 8, but the OTel span side was already lossy at 128/1000 events. Its median Arm C trace grew to 8,494,070 bytes because the retained events carried 64 KiB payloads, but the cell cannot support a "healthy through x1000 span events" claim.

The Slice 12 boundary result is:

• Kernel side: healthy through x1000 kernel events and concurrency 16 on this host class, at n=5 and without RSS collection. Artifact inspection found exactly 500/500 unique worker files in k500 and 1000/1000 in k1000, kc1000, and corner-lite, for both Arm A and Arm C, counted from event-rate-sweep/worker-* entries in extracted archive contents.
• Span side: first widened span cell (s500) is a trace-fidelity boundary under the default OTel JS SDK limits: 128/500 events retained. Arm A remains correctly asymmetric in these paired cells: it has no OTel trace export and records span pressure as baseline / 0.
• Corner side: combined kernel + span stress is bounded by the same span-fidelity limit, not by Runner health.

That closes the current event-rate arc for the default configuration. A future experiment can deliberately raise OTEL_SPAN_EVENT_COUNT_LIMIT and rerun the span cells, but that would be a new span-limit study, not a continuation of the default-config boundary sweep. That follow-up is tracked separately in issue #1408.

What This Means¶

• The delegated measurement harness is usable for all three arms: wall-clock and RSS runs produced all-valid samples on the same host class.
• The observed Arm C tail ratio is healthy for this deterministic workload on assay-bpf-runner.
• The observed Arm C median wall-clock is about 2x Arm B on the same host class for this workload, while RSS increases by about 7%.
• The observed Arm A and Arm C RSS medians are effectively identical at this scale, so the RSS delta versus Arm B is attributable to Runner capture rather than trace JSON export.
• Arm A's repeat wall-clock tail was healthy, but the later phase-timing run had an unhealthy tail and the runner-only median remained higher than Arm C, so wall-clock decomposition remains a caution, not a benchmark claim.
• Slice 8 phase timing localizes the largest measured internal phase delta to monitor attach, but the majority of the Arm A / Arm C median gap sits outside the current phase buckets.
• Slice 9 paired diagnostics show that the Slice 8 Arm A-over-Arm C median gap does not reproduce under adjacent pairing. Wall-clock residuals are close enough, and tails noisy enough, that the wall-clock decomposition should stop rather than spawning another broad rerun.
• Slice 11 shifts the useful overhead question from "which arm is faster?" to "where do health gates or artifact sizes start to scale?" At the starter matrix levels, the health boundary was not reached.
• Slice 12 extends that result: Runner kernel capture remained healthy through 1000 worker files and concurrency 16, while widened OTel span pressure hit the default SDK event-retention limit at the first widened span cell.
• The RSS path works on the delegated Linux runner with GNU /usr/bin/time -v; samples record the RSS tool version and emit peak_rss_bytes into both summary.json and the BMF export.
• The summary renderer now gives reviewers the same metrics in summary.md and in the GitHub step summary, while summary.json remains canonical.

What This Does Not Mean¶

• No product ranking is implied between OpenTelemetry, OpenInference, or Assay-Runner.
• No model/provider latency claim is made. The workload is deterministic and measurement-scoped.
• No co-temporal variance claim is made. Arm B and Arm C ran on the same host class at different times, and Arm A was dispatched separately as well.
• No additive wall-clock decomposition claim is made between "Runner archive only" and "Runner archive plus OTel trace". The phase-timing and paired residual runs show that the median gap is not stable under pairing, and the paired run has unhealthy tails.
• Slice 12 does not prove that 500 or 1000 OTel span events are safe or cheap under the default workload configuration. The trace retained only 128 span events at those targets, so timing above that point is fidelity-limited.
• No Trust Card or Trust Basis claim is added. This remains an experiment-scoped measurement follow-up.
• The generated artifacts remain review artifacts until a later decision explicitly promotes a measurement bundle into committed evidence.

Next Work¶

The correct publication language is now:

On the linux-aarch64-6.8.0-117-generic delegated runner host class, the current dual-capture path measured roughly +858 ms median wall-clock and +7.7 MB median RSS over OTel-only for this deterministic workload. The result is not co-temporal and does not decompose Runner archive-only cost.

Arm A measurements and phase-timing diagnostics have landed. The safe publication language is now:

On the same delegated host class, Arm A runner-only and Arm C dual-capture had effectively identical median RSS. The RSS decomposition points to Runner capture as the memory-cost source. Wall-clock decomposition remains inconclusive: phase timing explains part of the Arm A / Arm C median gap, mostly around monitor attach, but the majority remains outside the current Runner phase buckets and Arm A's phase run had an unhealthy tail.

Next engineering slice:

Do not add another broad Arm A/C wall-clock rerun for this arc. The paired residual diagnostic has landed and shows that the median gap is not stable enough for an additive wall-clock decomposition at the current measurement budget. Slice 12 has now produced the intended boundary statement: kernel capture stayed healthy through the widened kernel cells, and span widening hit the default OTel event-retention boundary at the first widened span cell.

The only logical follow-up is a new, explicitly scoped span-limit study: configure the OTel SDK span-event limit above the requested target, verify retained event counts first, and then rerun only the span cells needed to answer whether trace export cost scales after the fidelity boundary is removed. Track that as issue #1408, outside the closed default-config overhead arc.