Assay-Runner Phase 1 Spike Plan¶

Status: accepted on 2026-05-21 for the delegated Linux/eBPF Phase 1 path; internal spike record, not a public roadmap commitment Date: 2026-05-20 Scope: expands the Phase 1 proof spike from ASSAY-RUNNER-PRODUCT-CANDIDATE-2026-05-18.md; no external name, repo split, hosted service, or product build is implied.

This plan turns the Assay-Runner candidate memo into a bounded spike contract. It answers one question:

Can Assay produce one verifiable measured-run bundle per shim mode, with low-ambiguity layer correlation and honest observation health?

If the answer is no, this track stops.

Phase 1 Result¶

Phase 1 passed on delegated Linux/eBPF hardware on 2026-05-21.

The acceptance record is ASSAY-RUNNER-PHASE1-ACCEPTANCE-2026-05-21.md. It records workflow run 26211485614, commit 56571045, and gates=all passing for kernel-only, kernel+policy, and OpenAI Agents kernel+policy+SDK.

Non-Goals¶

Phase 1 must not become a hidden product launch. It explicitly does not include:

• public naming work beyond the internal Assay-Runner placeholder
• a new public repository
• a hosted service, dashboard, or production sidecar
• Phase 2 capability diff as a PR comment
• sequence semantics or causal ordering across layers
• event-level signatures or signed tool-call receipts
• cross-platform full measurement claims

The only deliverable is a local spike proving or disproving attribution.

Placement¶

Spike code should live in Rul1an/assay, not in a new repository.

Allowed source layout:

If a crate is added, its name must include spike and it must be publish-disabled. A hidden test harness is also acceptable if that proves the attribution question with less surface area. Do not create a publishable assay-runner crate before Phase 1 passes.

Existing Surfaces To Reuse¶

The spike should be mostly assembly:

Do not duplicate these semantics inside the spike unless there is no callable boundary yet.

Known integration gaps:

• extract_tool_call_id is not currently a stable public API for a separate spike crate. S4 may require a small assay-core adapter or visibility relaxation before correlation code can call it directly.
• Assay currently has more than one internal extract_tool_call_id function. Before S4 correlation work begins, choose one canonical extraction path for policy events and spike-side correlation, then cover that choice with a contract test.

Spike Output Contract¶

The spike writes one local archive:

assay-runner-spike-<run_id>.tar.gz

The archive should be accepted by assay evidence verify or by a clearly named temporary verifier if the existing bundle writer cannot carry every layer yet. If a temporary verifier is needed, the plan must record the gap before any Phase 2 work begins.

Minimum file shape:

manifest.json
events.ndjson
layers/kernel.ndjson
layers/policy.ndjson
layers/sdk.ndjson
capability-surface.json
observation-health.json
correlation-report.json

Allowed empty files:

• layers/policy.ndjson when policy_layer=absent
• layers/sdk.ndjson when sdk_layer=absent

No layer may be silently omitted.

Observation Health Contract¶

Phase 1 freezes this v0 shape:

{
  "schema": "assay.runner.observation_health.v0",
  "run_id": "run_...",
  "platform": "linux",
  "kernel_layer": "complete",
  "ringbuf_drops": 0,
  "policy_layer": "present",
  "sdk_layer": "absent",
  "cgroup_correlation": "clean",
  "notes": []
}

Allowed values:

Rules:

• ringbuf_drops > 0 forces kernel_layer=partial_ringbuf_drops.
• non-Linux runs force kernel_layer=absent.
• --agent-shim none forces sdk_layer=absent.
• an SDK shim may only set sdk_layer=self_reported unless kernel or policy evidence corroborates the specific event boundary.
• cgroup_correlation=failed is a failing Phase 1 run, even if the bundle verifies.

Capability Surface V0¶

Phase 1 summarizes kernel observations into deterministic sets:

{
  "schema": "assay.runner.capability_surface.v0",
  "run_id": "run_...",
  "filesystem_prefixes": ["/tmp/assay-runner-spike/"],
  "network_endpoints": ["127.0.0.1:8080"],
  "process_execs": ["/usr/bin/curl"],
  "mcp_tools": ["filesystem.read_file"],
  "policy_decisions": ["allow:filesystem.read_file"]
}

Serialization must be deterministic:

• sort every set lexicographically before writing JSON
• use stable path normalization rules
• use stable endpoint formatting: host_or_ip:port
• no timestamps in set entries
• no event ordering claims

This prevents byte-level diff noise before Phase 2 exists.

Correlation Model¶

Phase 1 correlation is set-based and window-based.

Required keys:

Timing rule:

• all runner-owned windows use one canonical clock source; on Linux, prefer a monotonic clock for joins
• SDK-shim timestamps are recorded as self-reported informational fields only and must not be used as the sole basis for kernel or policy attribution

Allowed claim:

Tool-call T occurred in window W, policy saw T, and kernel events K were
observed inside the measured cgroup during W.

Forbidden claim:

Tool-call T caused syscall S at timestamp N.

Workstreams¶

S0: Freeze The Spike Contract¶

Deliverables:

• this plan merged
• one follow-up issue or checklist for each workstream
• no code changes yet

Exit gate:

• second-party internal review agrees the spike is bounded and can be killed without product fallout; if no second party is available, record a written self-checklist in this note family before code work starts

S1: Runner Boundary And run_id¶

Implement the minimal orchestration boundary:

assay runner-spike run --agent-shim none -- <command...>

Hidden CLI is acceptable. A direct test harness is also acceptable if adding a CLI surface creates too much churn.

Responsibilities:

• generate one run_id
• create or identify one cgroup scope on Linux
• launch the child command inside that scope
• record window_start and window_end
• write an initial manifest and empty layer files

Acceptance:

• one command run produces one archive
• archive includes run_id in every file that has a run identity
• non-Linux produces a degraded bundle with kernel_layer=absent

S2: Kernel Layer And Health¶

Use assay-monitor rather than adding new probes.

Responsibilities:

• configure monitored cgroup
• listen to monitor events during the run window
• snapshot monitor stats before and after the run
• compute ringbuf_drops
• write layers/kernel.ndjson
• derive filesystem_prefixes, network_endpoints, and process_execs

Acceptance:

• a fixture that opens a known file records that file prefix
• a fixture that executes a known binary records that binary
• a fixture that connects to a local TCP endpoint records that endpoint where monitor support exists
• three repeated runs produce the same health metadata and same capability set for the deterministic fixture

S3: none + kernel-only¶

This is the epistemic proof path.

Fixture:

tests/fixtures/runner-spike/kernel-only-agent.sh

The fixture should:

• read one file under a temp directory
• write one file under a temp directory if write events are observable
• execute one stable binary such as /usr/bin/env
• optionally connect to a local listener

Acceptance:

• sdk_layer=absent
• policy_layer=absent
• kernel_layer=complete on Linux when no drops occur
• cgroup_correlation=clean
• the bundle verifies with assay evidence verify or, until runner-spike archives are carried by assay-evidence, with the temporary scripts/ci/runner-spike-kernel-only-acceptance.sh verifier
• three repeated fixture runs verify with scripts/ci/runner-spike-kernel-only-three-run-determinism.sh

S4: none + kernel+policy¶

This proves Runner still works when the agent uses Assay policy surfaces but does not use an SDK shim.

Fixture:

tests/fixtures/runner-spike/mcp-policy-agent.sh

The fixture should route one MCP tool call through assay mcp wrap or a minimal existing MCP proxy path.

Acceptance:

• sdk_layer=absent
• policy_layer=present
• policy event includes a stable tool_call_id
• kernel capability set is congruent with the allowed tool call
• correlation report joins policy to kernel by run_id, window, and cgroup
• the fixture verifies with scripts/ci/runner-spike-kernel-policy-acceptance.sh
• three repeated fixture runs verify with scripts/ci/runner-spike-kernel-policy-three-run-determinism.sh

S5: openai-agents Shim¶

This is adoption proof, not epistemic proof.

Transport is fixed for the spike:

• runner-spike launches node as a subprocess
• a small JS wrapper owns the @openai/agents SDK invocation
• the wrapper writes normalized SDK events as NDJSON to an explicit file or file descriptor supplied by runner-spike
• runner-spike consumes that stream and writes layers/sdk.ndjson
• stdout/stderr remain diagnostic channels; they are not the canonical SDK event stream

The first S5 implementation slice may ingest a prewritten normalized SDK event log through the same hidden runner boundary. That only freezes the assay.runner.sdk_event.v0 contract; it is not enough to claim the openai-agents shim has passed until the subprocess transport and deterministic fixture are wired.

The next S5 implementation slice proves the transport boundary with a deterministic SDK fixture:

tests/fixtures/runner-spike/sdk-event-wrapper.sh
scripts/ci/runner-spike-sdk-contract-acceptance.sh

Runner supplies ASSAY_RUNNER_SDK_EVENT_LOG, ASSAY_RUNNER_RUN_ID, and ASSAY_RUNNER_SDK_EVENT_SCHEMA to the measured subprocess. The subprocess writes normalized SDK NDJSON to that path; stdout and stderr remain diagnostic only.

The following S5 slice cross-checks SDK self-report against the existing policy layer:

tests/fixtures/runner-spike/sdk-policy-agent.sh
scripts/ci/runner-spike-sdk-policy-correlation.sh
scripts/ci/runner-spike-sdk-policy-three-run-determinism.sh
scripts/ci/runner-spike-sdk-policy-mismatch.sh
scripts/ci/runner-spike-sdk-policy-mismatch-three-run-determinism.sh

When a policy layer is present, every SDK tool-call id must match an existing policy correlation binding. A missing binding marks the correlation report partial with sdk_tool_call_without_policy_binding:<tool_call_id>. SDK events still do not create bindings or promote kernel/policy claims by themselves. The mismatch verifier intentionally emits a distinct SDK tool-call id and must observe that partial-correlation ambiguity across three deterministic runs.

The first real JavaScript SDK slice runs the same correlation gate through a Node subprocess that imports @openai/agents directly and uses a local deterministic model provider to force one read_file function-tool call:

runner-fixtures/openai-agents/fixture-agent.js
runner-fixtures/openai-agents/sdk-policy-agent.sh
scripts/ci/runner-spike-openai-agents-sdk-policy-correlation.sh
scripts/ci/runner-spike-openai-agents-sdk-policy-three-run-determinism.sh
scripts/ci/runner-spike-openai-agents-sdk-policy-mismatch.sh
scripts/ci/runner-spike-openai-agents-sdk-policy-mismatch-three-run-determinism.sh

This proves the @openai/agents runtime hook path and the runner-supplied SDK event stream without a live LLM request. It still is not the full delegated Linux kernel+policy+SDK acceptance gate: that requires the privileged eBPF host to run the kernel capture alongside the JavaScript shim. The OpenAI Agents mismatch verifier intentionally emits the SDK tool-call id tc_runner_sdk_only_001 while policy keeps tc_runner_policy_001, and must observe the same sdk_tool_call_without_policy_binding:<tool_call_id> ambiguity across three deterministic runs.

The full delegated-host S5 gate combines the real JavaScript SDK fixture with clean cgroup kernel capture and the MCP policy fixture:

scripts/ci/runner-spike-openai-agents-kernel-policy-acceptance.sh
scripts/ci/runner-spike-openai-agents-kernel-policy-three-run-determinism.sh
.github/workflows/runner-spike-delegated.yml

That gate is Linux/eBPF-only. It may skip on ordinary hosted CI, and S5 is not complete until this gate passes on a delegated cgroup host with kernel_layer=complete, policy_layer=present, sdk_layer=self_reported, cgroup_correlation=clean, empty correlation ambiguities, and stable health/surface/correlation/policy/SDK artifacts across three runs.

The delegated workflow is workflow_dispatch-only and runs on [self-hosted, linux, assay-bpf-runner] so ordinary PR CI cannot be blocked by runner availability. A skip from the underlying scripts is a failure in that lane: the delegated host is the place where Linux, cgroup v2, Node 22, and the eBPF artifact are expected to exist.

Keep it thin:

• emit normalized SDK events
• carry tool_call_id
• record SDK package/version when available
• never promote SDK events to verified side effects by themselves
• keep the wrapper killable with the measured process tree

Normalized SDK events:

tool_call_started
tool_call_completed
run_finished
run_failed

Determinism is mandatory:

• no live LLM calls are allowed in the spike suite
• S5 uses assay-core::vcr replay in strict mode, for example ASSAY_VCR_MODE=replay_strict, with a checked-in cassette or fixture
• if replay cannot be wired through @openai/agents, use a deterministic degenerate SDK fixture and record that blocker before claiming S5 pass

Acceptance:

• sdk_layer=self_reported
• tool_call_id matches policy-layer tool_call_id where MCP policy is used
• SDK events arrive through the subprocess NDJSON stream, not through ad hoc log scraping
• the S5 fixture runs in replay-strict or deterministic fixture mode
• kernel event set for the tool window is stable across three runs
• SDK-only data cannot make kernel_layer=complete

S6: Correlation Report¶

Write a machine-readable report:

{
  "schema": "assay.runner.correlation_report.v0",
  "run_id": "run_...",
  "status": "clean",
  "bindings": [
    {
      "tool_call_id": "tc_001",
      "policy_decision": "allow",
      "kernel_event_count": 3,
      "window": {
        "start": "2026-05-20T00:00:00Z",
        "end": "2026-05-20T00:00:01Z"
      }
    }
  ],
  "ambiguities": []
}

Status values:

• clean: all expected joins present
• partial: a layer is absent or a join is incomplete but declared
• failed: attribution is too ambiguous for the run to count

Acceptance:

• no successful Phase 1 demo may have status=failed
• ambiguity reasons are deterministic strings, not free-form diagnostics only

S7: Adversarial Spike Checks¶

Use assay-sim patterns or small local fixtures to stress the attribution model before declaring Phase 1 successful.

Required checks:

• child process forks before doing the file access
• child exits quickly after access
• two sibling processes touch different files in the same run
• policy-denied tool call produces a policy event without claiming matching allowed side effects
• induced or simulated ring-buffer pressure sets kernel_layer=partial_ringbuf_drops

Exit gate:

• if any normal deterministic fixture produces unstable attribution across three runs, Phase 1 fails

Scenario Matrix¶

Hard Kill Criteria¶

Stop the track if any of these are true after two implementation passes:

• cgroup correlation cannot be made clean for the raw deterministic fixture
• policy-to-kernel attribution is unstable across three repeated runs
• tool_call_id cannot be carried through the OpenAI Agents path without brittle SDK internals
• ordinary runs produce ring-buffer drops often enough that complete is not the normal Linux result
• bundle verification requires a parallel artifact system instead of an incremental extension to assay-evidence

If the track stops, write a short closure note instead of continuing into Phase 2.

Closure note path:

docs/notes/ASSAY-RUNNER-PHASE1-CLOSURE-<date>.md

The closure note is the sole deliverable when the track stops. It must record the failed criterion, the evidence observed, and whether any spike code should be deleted or left quarantined for later reference.

Success Criteria¶

Phase 1 passes only when all of the following are true:

• one none + kernel-only bundle verifies
• one none + kernel+policy bundle verifies
• one openai-agents bundle verifies or has a documented SDK blocker that does not affect the none proof path
• all three bundle types include observation health
• deterministic fixtures are stable across three repeated Linux runs
• adversarial checks do not break attribution silently
• all successful outputs are explicit about absent, partial, and self-reported layers

Passing Phase 1 does not authorize a public product launch. It only authorizes Phase 2 planning for capability diff.

Phase 2 Handoff, If Phase 1 Passes¶

The next plan should be ASSAY-RUNNER-PHASE2-CAPABILITY-DIFF-PLAN-*.

It must start from Phase 1 artifacts and answer:

• how capability surfaces are compared
• how ignore rules are expressed
• how Harness projects the diff
• which warnings block or annotate PR output
• which outputs are canonical JSON versus reviewer Markdown

Do not start Phase 2 until Phase 1 has real bundles checked into a fixture or artifact location.

Review Checklist¶

• The plan keeps Runner separate from Assay-Harness.
• The spike can be killed cleanly.
• none remains the first proof path.
• observation health is mandatory.
• deterministic serialization is required before diff work.
• no public naming, repo split, or external claims are introduced.