Assay-Runner Projection Roadmap¶

Status: planning reference, not a shipped contract. This page turns the runner-vs-OTel and cross-runtime drift experiments into a product-facing projection roadmap. It does not change the Runner archive v0 contracts, does not add a public CLI surface, and does not promote projection output to primary evidence.

Assay-Runner now has enough measured-run data to show a recurring product lesson:

Raw evidence is the source of truth. Projection reports make it readable.

The next useful work is not to claim more semantic equivalence. It is to make the boundaries between raw observation, classification, projection, and policy interpretation explicit enough that reviewers can tell which claim is being made.

SOTA Context¶

The relevant 2026 direction is layered observability:

Assay's product niche is the projection layer above measured runtime evidence: keep raw observed effects intact, then add narrow, auditable classification rows that say how a reviewer may compare them.

Reference anchors:

• OpenTelemetry GenAI semantic conventions: https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/
• OpenInference semantic conventions: https://arize-ai.github.io/openinference/spec/semantic_conventions.html
• Tetragon runtime observability: https://tetragon.io/docs
• Falco syscall field vocabulary: https://falco.org/docs/reference/rules/supported-fields/
• SLSA provenance model: https://slsa.dev/spec/v1.1/provenance

Core Rule¶

Every projection MUST preserve four things:

1. the raw observed value;
2. the projected value, if any;
3. the rule that produced the projection;
4. the confidence and non-claim attached to that rule.

Projection code MUST NOT silently rewrite raw evidence, infer semantic workload equivalence, or collapse unknowns into a convenient bucket.

Claim Levels¶

Projection output should use explicit claim levels instead of prose-only qualifiers:

Reports may include more detailed enums, but they should preserve this four-level ladder so readers can distinguish fact from projection.

Phase 1: Path Projection v0¶

Path projection is the highest-leverage next step. Current drift reports show too much absolute run-directory noise. The fix is not to remove raw paths; it is to add a logical path layer above them.

The smallest buildable first slice of this phase — declared-rule projection only, no heuristics, no network — is scoped in path-projection-slice1-plan.md.

Shape¶

{
  "raw_path": "/opt/actions-runner/_work/assay/.../workdir/fixture-input.txt",
  "projected_path": "workdir/input",
  "path_class": "workload_fixture",
  "relation": "inside_run_workdir",
  "rule": "workload_contract_input_path",
  "confidence": "declared",
  "claim_level": "projected_equivalent"
}

Initial path classes¶

Rules¶

• Workload-contract paths have the highest confidence: declared.
• Run workdir prefix stripping is allowed only when the prefix is declared by the workflow or artifact metadata.
• Prefix or substring heuristics must emit confidence=heuristic.
• Unknown paths remain unknown; they are not failures by default.

Acceptance Criteria¶

• Raw path sets remain present in the report.
• Every projected path carries rule and confidence.
• Projection mappings stay compact: declared mappings are explicit, unmatched raw values are summarized by count and samples, and raw row sets remain the source of truth.
• Projection is idempotent.
• A report can show "raw differs, projected matches" without calling the workloads semantically equivalent.

Phase 2: Runtime / Noise Taxonomy v0¶

The taxonomy should be designed in parallel with path projection because path and network projection both need the same classification language.

Implementation note: the cross-runtime drift comparator now emits this taxonomy as vocabulary-only metadata. It validates declared projection classes and preserves unknown, but it does not yet infer taxonomy classes from paths or endpoints.

Initial taxonomy:

Taxonomy strings are reviewer vocabulary, not policy verdicts. A provider_sdk difference may be expected, suspicious, or irrelevant depending on the policy layer that consumes the report.

Phase 3: Report Provenance Metadata¶

Report-level provenance should land early because it improves every later experiment at low implementation cost.

Implementation note: the cross-runtime drift comparator now emits a provenance block in JSON reports. Archive manifest digests, schema versions, observation-health gates, and correlation status are derived from the two input archives. Workflow URL, runner label, kernel tuple, Assay version/capture commit, render metadata, and eBPF object digest are caller-supplied anchors. render_metadata.kind is always present and defaults to unspecified; the other optional render metadata fields stay null when not provided.

Minimum metadata block:

{
  "assay_version": "3.x.y",
  "assay_commit": "sha",
  "render_metadata": {
    "kind": "live_capture | re_render | synthetic_fixture | unspecified",
    "rendered_at": "2026-05-25T18:55:39Z",
    "comparator_commit": "sha",
    "source_run_url": "https://github.com/Rul1an/assay/actions/runs/...",
    "raw_captures_unchanged": true
  },
  "runner_schema_versions": {
    "archive": "assay.runner.archive.v0",
    "kernel_event": "assay.runner.kernel_event.v0"
  },
  "kernel": {
    "os": "linux",
    "release": "6.8.0-117-generic",
    "arch": "aarch64"
  },
  "ebpf_object_digest": "sha256:...",
  "workflow": {
    "url": "https://github.com/Rul1an/assay/actions/runs/...",
    "runner_label": "assay-bpf-runner"
  },
  "input_archives": [
    {
      "run_id": "run_arm_a_...",
      "manifest_digest": "sha256:..."
    }
  ]
}

The metadata block is for anchoring and reproducibility. It does not upgrade a projection into primary evidence.

Phase 4: Network Projection v0¶

Network projection should follow path projection. Keep the taxonomy small and allow unknown.

Implementation note: the cross-runtime drift comparator now supports declared exact endpoint aliases and CIDR aliases. Raw endpoints remain unchanged in the report; projection is additive and unmatched endpoints stay unknown.

Shape¶

{
  "projection": {
    "schema": "assay.runner.network_projection.v0",
    "status": "applied",
    "dimension": "network_endpoints",
    "claim_level": "projected_equivalent",
    "in_both": ["provider_api"],
    "rules": ["declared_network_cidr_alias"],
    "mappings": [
      {
        "side": "a",
        "raw_value": "34.120.10.20:443",
        "projected_value": "provider_api",
        "network_class": "provider_api",
        "relation": "declared_cidr",
        "rule": "declared_network_cidr_alias",
        "confidence": "declared",
        "claim_level": "projected_equivalent"
      }
    ]
  }
}

Initial endpoint classes¶

IP-only endpoints should remain unknown unless the report can name the classification rule. DNS, SNI, SDK metadata, or an explicit allowlist may raise confidence, but the report must say which source was used.

Phase 5: Runtime Drift Projection Artifact¶

Path projection, network projection, taxonomy, and metadata are now stable enough to freeze the first projection artifact shape.

Implemented schema:

assay.runner.runtime_drift.v0.2

This is a projection artifact, not a primary evidence artifact. It reads Runner archives and emits a comparison report. The contract is documented in runtime-drift-v0.md, with a machine-readable schema at schema/runtime-drift-v0.2.schema.json.

Required design properties:

• input archive manifest digests are recorded;
• raw and projected views are both present;
• every projection row carries rule and confidence;
• health gates are copied from the source archives;
• non_claims are machine-readable;
• policy acceptability is out of scope.

Example row:

{
  "dimension": "kernel_file_operations",
  "classification": "runtime-induced",
  "claim_level": "projected_equivalent",
  "reason": "raw absolute paths differ; projected fixture operations match",
  "raw": {
    "base_only": ["open_read:/tmp/a/workdir/fixture-input.txt"],
    "head_only": ["open_read:/tmp/b/workdir/fixture-input.txt"]
  },
  "projected": {
    "unchanged": ["open_read:workdir/input"]
  },
  "rules": ["workload_contract_input_path"],
  "non_claims": [
    "projection_no_semantic_workload_equivalence",
    "projection_no_policy_acceptability_verdict"
  ]
}

Phase 6: Kernel Event Metadata Schema¶

Kernel expansion is valuable, but it should come after the projection layer is readable. That condition is now met by runtime-drift-v0.md. The first kernel follow-up is deliberately small: freeze the current enriched kernel_event.v0 line shape in schema/kernel-event-v0.schema.json so consumers can validate whether an archive supports operation-aware open projections.

The open metadata added to kernel event v0 already supports operation-aware projections for openat and openat2 calls. Successful calls can project access and operation hints:

• read;
• write;
• read_write;
• create;
• truncate;
• append.

Failed calls are represented separately by the kernel event status field (status=error) derived from the syscall return value. error is not an operation class alongside read or write.

Future kernel-event expansion should consider:

Do not add fd-level byte semantics until the report can keep them separate from open-intent semantics. open_read is not the same claim as "bytes were read from this file."

Non-Claims¶

Projection reports MUST carry explicit non-claims. Initial codes:

Suggested Slice Sequence¶

Relationship To Existing Contracts¶

• artifacts-v0.md remains the archive evidence contract. Projection output reads those artifacts; it does not replace them.
• cross-runtime-diff-v0.md remains the narrow Phase 2C capability-surface projection with work-dir prefix canonicalization only.
• The cross-runtime drift experiment is an exploratory implementation proving why richer projection layers are useful.