PR Gate Output Contracts Specification v1¶

Status: Draft Version: 1.0.1-draft Date: 2026-02 ADR: ADR-019: PR Gate 2026 SOTA Related: DX-IMPLEMENTATION-PLAN, SPEC-GitHub-Action-v2.1

1. Overview¶

This specification defines the output contracts for the Assay PR gate: the blessed flow outputs (junit.xml, sarif.json, summary.json), exit and reason code semantics, SARIF constraints for GitHub compatibility, and the requirement that every non-zero exit provides a suggested next step. Implementations of assay ci and assay run (when used as the CI entrypoint) MUST conform to this spec so that CI consumers and the GitHub Action get predictable, machine-readable results.

Design Principles¶

• PR-native — Outputs integrate with GitHub (JUnit → test annotations, SARIF → Security tab, Check Run Summary) without custom glue.
• Stable and versioned — summary.json carries a schema_version so consumers can detect and adapt to changes.
• Machine-readable nuance — Exit codes stay coarse (0/½/3); reason codes in summary.json and console provide stable, fine-grained semantics without breaking exit-code scripts.
• Upload-safe — SARIF stays within GitHub limits (size, result count) so upload never fails randomly; every result has at least one location.

2. Blessed Flow Outputs¶

When assay ci (or the equivalent run invoked by the blessed workflow) completes, it MUST produce the following artifacts in the configured output directory (default: .assay/reports or equivalent).

Normative: The blessed entrypoint is assay ci. The same three outputs MUST be produced so that one local command reproduces exact CI behaviour.

3. summary.json Schema¶

3.1 Required Top-Level Fields¶

3.2 Provenance (Artifact Auditability)¶

Every summary.json MUST include a top-level provenance object with the following fields so that gates remain auditable (ADR-019 P0.4).

Normative: If the run was executed with --no-verify, verify_mode MUST be "disabled". When replay is used, producers SHOULD set replay=true and include bundle_digest and replay_mode.

3.3 Results Summary (Optional but Recommended)¶

A top-level results object MAY contain:

A top-level performance object MAY contain total_duration_ms (integer, milliseconds). Future versions MAY add slowest_tests, cache_hit_rate, phase_timings (see ADR-019 / DX-IMPLEMENTATION-PLAN). Consumers MUST ignore unknown top-level keys.

3.3.1 Seeds (E7.2 – Replay Determinism)¶

A top-level seeds object (summary.json) and top-level seed_version, order_seed, judge_seed (run.json) SHALL be present for schema stability. On early-exit (e.g. trace not found, config fail), seeds may be null when unknown; seed_version SHALL still be present.

Normative: run.json (extended and minimal) and summary.json SHALL include seed_version; order_seed and judge_seed SHALL be present (string or null). Seeds MUST be encoded as decimal strings (or null) to avoid precision loss in JSON consumers (e.g. JS/TS safe for u64 > 2^53). CLI console SHALL print one line: Seeds: seed_version=1 order_seed=… judge_seed=… so CI job summaries can show them for replay.

3.3.2 Judge Metrics (E7.3)¶

When the run had judge evaluations, a top-level judge_metrics object MAY be present with low-cardinality reliability metrics:

Normative: Judge unavailable (transport/infra) MUST NOT be counted as Abstain; use unavailable_count for that.

Implementation note (unavailable_count): Implementations may use message heuristics (e.g. timeout, 5xx, rate limit, network) on Error-status rows to classify infra failures. Abstain (uncertain verdict) is never counted as unavailable. Prefer standardised reason codes or an explicit infra_class field when available.

Implementation note (flip_rate): The spec defines flip_rate as “order was swapped and outcome differed”. When the judge does not record whether the pass/fail verdict would have differed under the other ordering, implementations may use a heuristic proxy (e.g. swapped and non-unanimous agreement). This proxy does not guarantee that the verdict actually flipped; it indicates order may have affected the outcome. When present, run.json and the CLI console SHALL expose judge metrics so CI can display them.

3.4 Example (Minimal)¶

{
  "schema_version": 1,
  "reason_code_version": 1,
  "exit_code": 0,
  "reason_code": "",
  "provenance": {
    "assay_version": "2.12.0",
    "verify_mode": "enabled"
  },
  "results": {
    "passed": 10,
    "failed": 0,
    "total": 10
  },
  "performance": {
    "total_duration_ms": 1234
  }
}

3.5 Example (Non-Zero with Next Step)¶

{
  "schema_version": 1,
  "reason_code_version": 1,
  "exit_code": 2,
  "reason_code": "E_TRACE_NOT_FOUND",
  "message": "Trace file not found: traces/ci.jsonl",
  "next_step": "Run: assay doctor --config ci-eval.yaml --trace-file traces/ci.jsonl",
  "provenance": {
    "assay_version": "2.12.0",
    "verify_mode": "enabled"
  }
}

4. Exit Code Registry¶

Exit codes are coarse and MUST NOT be redefined in a breaking way. Reason codes (§5) carry the nuance.

Normative: Judge failures (rate limit, provider 5xx, timeout) MUST map to exit code 3. Behaviour for security vs quality suites is policy-driven (fail-closed vs degrade/skip) per ADR-003/ADR-004; the exit code alone does not change.

Compatibility: Historically, some documentation used exit 3 for "trace file not found". Under this spec, trace-not-found is exit 2 with reason_code E_TRACE_NOT_FOUND. Implementations MAY support a compatibility mode (e.g. --exit-codes=v1) that preserves the old mapping for a documented deprecation period.

5. Reason Code Registry¶

Reason codes are stable, machine-readable strings. CI and scripts MAY branch on reason_code in summary.json. New codes MUST be added in a backward-compatible way (new string values); existing codes MUST NOT be removed or repurposed without a schema_version bump and migration notes.

5.1 Config / User Error (exit_code 2)¶

5.2 Infra / Judge Unavailable (exit_code 3)¶

5.3 Test Failure (exit_code 1)¶

Test-level failures MAY use existing policy/metric codes (e.g. E_ARG_SCHEMA, E_SEQUENCE_VIOLATION) or a generic E_TEST_FAILED. The summary.json reason_code for the run MAY be E_TEST_FAILED when at least one test failed and no single dominant reason is reported.

Normative: When exit_code ≠ 0, summary.json MUST set reason_code to one of the registered values (or a documented extension). Implementations MUST NOT leave reason_code empty when exit_code ≠ 0.

6. SARIF Contract (GitHub Compatibility)¶

SARIF produced for GitHub Code Scanning MUST satisfy the following so that upload-sarif does not reject the file.

6.1 Schema and Version¶

• SARIF version MUST be "2.1.0".
• Schema URI MUST be the official SARIF 2.1.0 JSON schema.

6.2 Location Requirement¶

• Every result MUST have at least one location. If no file/line is available, the producer MUST emit a synthetic location (e.g. URI assay.yaml, policy.yaml, or the config path). GitHub's upload can fail with "expected at least one location" when a result has an empty locations array.

Normative: Contract tests MUST validate that every result in the generated SARIF has locations length ≥ 1.

6.3 Truncation (Size and Result Limits) (E2.3)¶

• GitHub enforces limits on SARIF upload (e.g. max size gzipped, max number of results). Producers MUST truncate results when limits would be exceeded, and MUST add a clear indication that results were omitted (e.g. in the run description or a dedicated message: "N results omitted due to GitHub upload limits").
• Truncation strategy: keep top N results by severity (e.g. error first, then warning). N and the exact message are implementation-defined but MUST be documented. Truncation MUST be deterministic (same run → same selection); selection order: blocking (Fail/Error) first, then warning-level, then stable sort (e.g. by test_id). Eligibility: only SARIF-eligible results (e.g. Fail, Error, Warn, Flaky, Unstable) count toward the limit. Define eligible_total as the count of SARIF-eligible results before truncation; included as the number of results actually written in the SARIF run; then omitted_count = eligible_total − included.
• SARIF run-level metadata when truncated: runs[].properties.assay MUST be present when truncation was applied:
• truncated (boolean): true
• omitted_count (integer): number of eligible results omitted
• summary.json and run.json — nested sarif object: When SARIF was truncated, summary.json and run.json MAY include a top-level sarif object. Schema when present:
• sarif (object, optional): present only when truncation occurred (recommended to reduce noise).
• sarif.omitted (integer, required when sarif is present): ≥ 1.
• Consistency: When both are present, sarif.omitted (in run.json or summary.json) MUST equal runs[0].properties.assay.omitted_count.
• Normative: SARIF upload MUST NOT fail due to size or result count; truncation is required when necessary. Consumers MUST treat SARIF as potentially truncated and MUST use summary/run for authoritative counts.

6.4 Severity Mapping¶

• Map Assay outcomes to SARIF severity: Fail/Error → "error"; Warn/Flaky → "warning"; Info/other → "note".

7. Next-Step Requirement¶

For every non-zero exit, the implementation MUST provide at least one suggested next step so that users and CI logs know what to do next.

• Console: When exiting with exit_code ≠ 0, the process MUST print at least one line that is a concrete command or hint (e.g. "Run: assay doctor ...", "See: assay explain ...", "Fix baseline: assay baseline record ...").
• summary.json: The next_step field SHOULD be set when exit_code ≠ 0 (see §3.1). It MAY be the same as or a shortened form of the console message.

Normative: Contract tests MAY verify that for a set of known error conditions (missing config, missing trace, failing test), the output contains a non-empty next_step (in summary.json) and a console line with a suggested command.

8. Conformance¶

• Producers: assay ci and any code path that writes summary.json, junit.xml, or sarif.json for the PR gate MUST follow §2–§7.
• Consumers: CI workflows and the GitHub Action MAY rely on schema_version, exit_code, reason_code, and next_step as defined above. Unknown summary fields MUST be ignored.
• Contract tests: Implementations MUST include tests that (1) validate summary.json schema_version and required fields, (2) validate that every SARIF result has at least one location, (3) optionally validate SARIF against the official 2.1.0 schema and/or a minimal upload-smoke test.

9. Version History¶

10. References¶

• ADR-019 PR Gate 2026 SOTA
• DX-IMPLEMENTATION-PLAN
• SARIF 2.1.0
• GitHub Code Scanning SARIF