Skip to content

Assay-Runner Capability-Diff Phase 2B Plan

Internal Phase 2B planning note. This page defines the first capability-diff contract slice to design after the Phase 2A runner contracts. It is not a schema contract, not a CLI design, and not a product promise.

Phase 1 proved that the delegated runner can produce deterministic normalized evidence for one Linux/eBPF OpenAI Agents fixture. Phase 2A made that proof reviewable through artifact, fixture, CI-lane, proof-pack, and boundary contracts.

The first Phase 2B capability-diff slice should answer a narrower question:

Given two clean normalized runner evidence sets, what capability surface
changed, and what evidence supports that projection?

The first slice must stay deliberately small. It should freeze the diff contract before adding a broad diff engine, second runtime, macOS path, or public reporting surface.

The promoted v0 contract is capability-diff-v0.md. It intentionally chose a surface-level shape rather than the earlier draft per-binding shape because the current v0 artifacts do not carry per-binding capability values.

Inputs

The v0 diff contract should consume normalized runner artifacts, not raw telemetry:

Input Role
observation-health.json Determines whether a diff may be clean, partial, or failed
capability-surface.json Provides normalized observed capability sets
correlation-report.json Provides stable binding identity through tool_call_id

The following may be retained for diagnostics, but they are not primary v0 diff inputs:

  • raw kernel telemetry
  • workflow logs
  • proof-pack metadata
  • layers/sdk.ndjson
  • layers/policy.ndjson

Proof packs are useful for durable review and future reproduction, but the first diff contract should remain a pure projection over normalized evidence.

Fixed Decisions

These decisions are inherited from the Phase 2A contracts:

  • clean v0 diff requires kernel_layer=complete, ringbuf_drops=0, policy_layer=present, sdk_layer=self_reported, and cgroup_correlation=clean
  • clean SDK-to-policy binding requires stable tool_call_id
  • call-id-less runtimes are out of scope for the first diff contract
  • SDK timestamps are informational only and must not become primary join keys or ordering fallbacks
  • filesystem_paths contains full observed path values; prefix grouping is a later projection choice, not a rewrite of the artifact contract
  • the normalizer owns evidence boundaries; capability diff consumes those boundaries and must not broaden them

Diff Basis

The first contract should compare observed capability surfaces. It should not claim a full declared-versus-observed gap until there is a versioned declared capability input.

For the first slice, the safe vocabulary is:

Category Meaning
observed Present in normalized capability evidence
policy_observed Present in normalized policy decision summaries, including allow or deny outcomes
bound Connected to a clean tool_call_id correlation binding
unbound Present in normalized evidence but not connected to a clean binding

The contract may describe added, removed, and unchanged capability values between two evidence sets. It must not decide whether a change is acceptable; that remains policy, reviewer, or Harness responsibility.

Output Shape Direction

The v0 contract should choose the smallest shape the current artifacts can honestly support:

  • surface-level added, removed, and unchanged sets
  • binding-id stability by tool_call_id
  • explicit preconditions separate from scope claims
  • explicit unbound buckets without inventing per-binding path attribution

The contract must not claim per-binding filesystem, process, endpoint, or tool diffs until a versioned input supplies per-binding capability values.

Status Semantics

The promoted v0 contract defines five states:

Status Rule
clean Both input evidence sets have complete health, sdk_layer=self_reported, clean correlation, internally consistent run ids, and stable binding ids
partial:health At least one input can be parsed but has incomplete health such as ring-buffer drops or incomplete cgroup correlation
partial:correlation Health is sufficient to parse, but at least one correlation report is partial, ambiguous, or lacks stable binding identity
partial:unbound Reserved for a future per-binding capability artifact; v0 producers must not emit this status from run-global capability-surface values
failed Required artifacts are missing, schema strings are unsupported, run ids are inconsistent, or binding identity is unusable

ringbuf_drops > 0 must not be softened into a clean diff. A partial diff may help triage a failed run, but it must remain visibly incomplete.

Non-Goals

Do not include the following in the first slice:

  • second runtime support
  • call-id-less order fallback
  • macOS runner support
  • OTel or GenAI semantic-convention mapping
  • raw telemetry diffing
  • proof-pack ingestion as a required input
  • public reporting language
  • extraction into a separate runner repository
  • automatic merge or branch-protection changes

Contract PR Acceptance

The follow-up contract PR should:

  1. Add a versioned capability-diff contract page under docs/reference/runner/.
  2. Add at least one golden or shape example for the diff output.
  3. State exactly which normalized artifacts are inputs.
  4. Reuse the accepted S5 fixture as the first clean example.
  5. Keep tool_call_id required for clean binding identity.
  6. Preserve the ringbuf_drops=0 clean-diff rule.
  7. Avoid runtime behavior changes unless the PR also updates delegated proof.

Docs-only contract work can use ordinary docs CI. Any PR that changes runner artifact assertions, acceptance scripts, or executable diff code must follow the CI lane contract.

Open Questions

These should be answered by the contract PR, not by this mini-plan:

  • Should path projection group filesystem_paths by prefix, exact path, or both?
  • Does v0 diff include process and network categories, or start with filesystem plus MCP/tool categories only?
  • Where does a future declared-capability input live? This remains outside v0 and belongs in a separate Phase 2C+ contract slice.
  • Is the first implementation a Runner helper, a Harness projection, or an Assay-side reference checker?
  • Should unbound evidence be represented only per category for v0, or can a future input safely support per-binding-window unbound evidence?

The boundary map currently places capability-diff projection semantics on the Trust Basis / Harness side while Runner delivers clean input bundles. The contract PR may confirm or revisit that placement, but it must not silently relocate the boundary.

Suggested Sequence

  1. Land this mini-plan.
  2. Draft capability-diff-v0.md with one golden shape.
  3. Add read-only validation: diff(S5_acceptance, S5_acceptance) must produce status=clean with zero added, removed, or unbound entries, and the output must validate against the v0 contract schema.
  4. Implement a narrow projection only after the contract review settles.
  5. Revisit second-runtime fixtures after the diff contract has one clean consumer. Entry discipline for that line is defined in second-runtime-plan.md; cross-runtime capability-diff (e.g. diff(second_runtime, S5)) is intentionally out of Phase 2B scope. The Phase 2C cross-runtime mini-plan that follows from that boundary is in cross-runtime-diff-plan.md; its interpretation decisions are recorded in cross-runtime-diff-decisions.md.

Step 3 is implemented by scripts/ci/assay_runner_capability_diff_validate.py. Step 4 starts by letting the same helper project v0 diffs from explicit base/head normalized evidence directories. It remains an Assay-side reference projection over normalized artifacts rather than a runner execution path.