DX Implementation Plan — Default Gate Readiness¶

Status: Living plan (updated after Wave A merge) Date: 2026-02-08 Source: Critical DX review of DX-REVIEW-MATERIALS.md; aligns with ADR-019 PR Gate 2026 SOTA and ROADMAP. Aangepast na SOTA/DX reality check: technische correcties (GitHub Actions ref, SARIF limits, exit-codes compat), P0 Go/No-Go checklist, scope trims (E6a/E6b, cost guardrails, scrubbing deny-by-default). Score na aanpassingen: 9.7/10.

This document turns the DX review into a concrete backlog with per-file patchlist and test cases. Work is ordered P0 (must-have before default gate) then P1 (SOTA).

RFC-001 Execution Track¶

Canonical RFC for debt-ranked execution: - RFC-001: DX/UX & Governance

PR order for the new track: 1. PR-A1: typed error boundary + centralized reason-code mapping (Wave A start). 2. PR-A2: remove strict-mode env mutation (set_var) in run/ci path. 3. PR-A3: canonical config writing hardening (init/templates + docs). 4. PR-B1/B2/B3: pipeline unification + coupling reduction + --pack to --preset. 5. PR-C*: perf/scale only when benchmark data justifies it.

Current blocker gates (re-assessed on implemented code): - Wave A blocker: A1 must become truly typed at classification boundary (stable fields first, substring fallback explicit/legacy only). - Wave A blocker: A1 boundary errors need stable forensic fields (path/status/provider) to avoid message-only support triage. - Wave B blocker: B1 requires explicit run-vs-ci parity contract tests for exit/reason and output invariants. - P2 alerts (non-blocking): replay coupling wording update, A2 scope clarity (run/ci vs CLI-wide), B3 deprecation timeline as governance.

Current branch focus: - PR-A1 (merged to main via #198): typed boundary mapping for run/ci hot-path triage with unit coverage. - PR-A2/A3 (merged to main via #202): strict-mode env mutation removal + canonical init/template config writing. - PR-B1/B2/B3 (merged to main via #204/#205/#209): pipeline unification + dispatch decoupling + --preset rename with compat aliases. - Wave C kickoff: - PR-C0 (#212, open): additive performance trigger metrics + Wave C trigger guardrails in RFC-001. - PR-C1 (#213, open): reproducible verify/lint perf harness + workload budgets (docs/PERFORMANCE-BUDGETS.md). - PR-C2 (#214, open): runner clone overhead measurement surfaced in summary performance metrics. - PR-C3 (current branch): profile-store harness + runtime load/merge/save telemetry and trigger warnings. - PR-C4 (next): bounded run-id digest tracking beyond short ring buffer + memory/eviction visibility.

P0/P1 Epic Execution Summary¶

Compact execution view for all P0/P1 workstreams.

Recommended sequence: 1. EP1-1 GitHub Action v2.1 (compliance-pack support). 2. EP1-2 Golden Path (<30m first signal). 3. EP1-3 + EP1-4 parity hardening (docs/examples/contract tests; no feature expansion). 4. EP1-5 Watch hardening (determinism + Windows/file edge cases + loop tests). 5. EP1-6/EP1-7/EP1-8 parallel where capacity allows.

Explicit deferred boundaries: - no native notify watcher backend now; - no full-repo docs link checker as hard CI gate; - no non-Unix atomic-write parity expansion in this slice; - no dedicated IDE governance control-plane in this phase.

No-Regression Gates (Permanent)¶

Gate A (contract stability): - run.json / summary.json contracts, SARIF/JUnit outputs, and GitHub Action I/O remain backward-compatible by default.

Gate B (onboarding velocity): - clean repo -> first actionable Assay signal remains under 30 minutes on documented golden path.

Any P1 epic that violates A or B must either: - include an explicit migration plan, or - be split so contract/onboarding stability lands first.

P1 DX Contract Surfaces¶

Per epic we define what is normative (stable contract) versus best-effort (implementation detail).

EP1-1 Action v2.1 (compliance-pack first)¶

• Normative:
• compliance-pack resolution behavior and logged resolved pack reference.
• distinct failure modes: missing pack vs invalid pack vs lint/policy fail.
• output parity across Action surfaces (summary/SARIF/JUnit).
• Best effort:
• internal caching strategy for pack resolution.
• non-contractual log phrasing.

EP1-2 Golden Path (<30m first signal)¶

• Normative:
• documented bootstrap flow must produce an actionable first signal.
• generated scaffold commands in docs must execute as written.
• regression gate enforces onboarding time budget.
• Best effort:
• exact sample fixture contents.
• cosmetic scaffold formatting.

EP1-3 Explain + Compliance Hints (in review)¶

• Normative:
• --compliance-pack behavior and compatibility with non-pack mode.
• article hint + coverage summary field presence in supported output modes.
• failure output includes concrete next-action guidance.
• Best effort:
• wording of explanatory prose.
• ordering of non-contractual detail lines.

EP1-4 Drift Visibility (generate --diff) (in review)¶

• Normative:
• stable added/removed/changed semantics for drift output.
• deterministic diff output for identical inputs.
• --diff does not alter existing write semantics without explicit write flags.
• Best effort:
• pretty-print formatting and grouping style.
• optional metadata lines.

EP1-5 Watch hardening (existing command, hardening only)¶

• Normative:
• debounce clamp range and trigger-coalescing behavior.
• watch-loop exit semantics (loop lifecycle vs run result logging).
• config parse failure fallback: keep watching at least config/trace/baseline.
• Best effort:
• polling interval tuning.
• filesystem timestamp granularity handling nuances.

EP1-6 Privacy-safe observability defaults¶

• Normative:
• safe-by-default redaction and cardinality guardrails are on by default.
• unsafe raw prompt/body exposure requires explicit opt-in configuration.
• default exports do not leak prompt/response bodies.
• Best effort:
• exact redaction text tokenization strategy.
• non-contractual telemetry attribute ordering.

EP1-7 MCP auth hardening (E6a)¶

• Normative:
• RFC 8707 resource/audience constraints enforced.
• JWT alg/typ/crit validation and JWKS rotation behavior enforced.
• no-pass-through token behavior enforced.
• Interop matrix (required):
• JWKS rotation / kid miss.
• alg confusion + typ/crit rejection.
• audience/resource mismatch handling.
• Best effort:
• cache refresh cadence internals.
• diagnostics verbosity.

EP1-8 Replay bundle hardening¶

• Normative:
• verify/scrub defaults are safe and on by default.
• bundle manifest captures deterministic replay-critical metadata.
• unsafe/raw capture paths require explicit opt-in.
• Best effort:
• archive layout details that do not affect verification/replay contract.
• optional manifest annotation fields.

Progress Update (2026-02-08)¶

Recent implementation state:

• Wave A merged to main:
• #198 (A1): centralized run/ci error classification via typed boundary helpers.
• #202 (A2/A3 integration): strict-mode env mutation removal + canonical scaffold/config writing.
• Wave B merged to main (B1/B2):
• #205: shared pipeline + coupling reduction landing path.
• Wave B3 in final integration:
• #206 merged into codex/rfc001-wave-b2-coupling.
• #209 open (codex/rfc001-wave-b2-coupling -> main) with auto-merge enabled.
• P0/P1 DX slices merged earlier to main:
• docs/CLI parity, doctor --fix, watch hardening, Action v2.1 pack contracts, and follow-up parity checks.
• Deferred by design (unchanged):
• native notify backend,
• full-repo docs link checks as hard gate,
• cross-platform atomic-write parity beyond Unix.

Roadmap-aligned next execution order from here: 1. Land #209 to complete Wave B3 on main. 2. Start Wave C0 (perf control-plane instrumentation and CI baseline artifacts). 3. Execute C1-C4 only when C0 metrics cross trigger thresholds.

Explicit "do not implement now" decisions: - Do not migrate to a native notify watcher yet (keep dependency-free polling in place). - Do not switch to full-repo docs link validation yet (keep changed-files guard). - Do not broaden doctor atomic-write guarantees beyond Unix in this slice. - Do not add a dedicated IDE governance control plane yet (focus on CLI/CI/PR surfaces first).

Wave C Execution Blueprint (SOTA 2026)¶

Wave C is optimization-only and must stay contract-safe.

C0 first: metrics and gates (required)¶

Before C1-C4, add a stable perf signal surface:

• summary.json perf fields:
• verify_ms, lint_ms, runner_clone_ms, profile_store_ms, run_id_memory_bytes.
• CI artifact baseline (bench-baseline.json) for PR compare.
• PR perf gate classes:
• informational drift (non-blocking),
• threshold regression (blocking with explicit reason).

C1-C4 trigger table¶

Wave C hard stop-lines¶

• No C-task without a referenced C0 measurement snapshot.
• No optimization that changes run/summary/SARIF/JUnit contract shape without versioning.
• No optimization that weakens determinism or evidence integrity guarantees.

Post-#191 Follow-up Plan¶

After integration PR #191 lands in main, execution continues in three narrow follow-up slices to avoid scope creep:

1. PR A: init hello-trace colocation
2. Branch: codex/p1-init-hello-trace-colocation
3. Change: make assay init --hello-trace write traces/hello.jsonl relative to the directory of --config.

4. Acceptance:

   • assay init --hello-trace --config /tmp/x/eval.yaml creates /tmp/x/traces/hello.jsonl.
   • Existing default flow remains unchanged for local eval.yaml.

5. PR B: doctor dry-run exit contract

6. Branch: codex/p1-doctor-dry-run-exit-contract
7. Change: align doctor --fix --dry-run exit codes with documented diagnostics contract.

8. Acceptance:

   • Dry-run still writes nothing.
   • Exit code semantics are explicit and consistent across code, tests, and docs.
   • doctor_fix_e2e expectations match the final contract.

9. PR C: watch RunArgs drift reduction (optional)

10. Branch: codex/p1-watch-runargs-builder
11. Change: reduce/manual RunArgs duplication in watch execution path to avoid default drift over time.
12. Acceptance:
    • No behavior change in watch output/exit semantics.
    • Refactor is covered by existing watch/run tests.

Delivery guardrails for all three follow-ups: - Keep slices independent and reviewable. - Do not change run/summary/action output contracts unless explicitly intended and documented. - Update docs/DX-ROADMAP.md status immediately after each merge.

EU AI Act date anchors used in this plan: - 2025-02-02: first phased obligations active. - 2025-08-02: GPAI-focused obligations active. - 2026-08-02: broader obligations active.

DX North Star (2026)¶

Use this scorecard as a gate for roadmap choices. If a new item does not clearly improve at least one dimension below, it is de-prioritized.

Execution Filters¶

• Prefer paved-road improvements over adding new interfaces.
• Keep policy gate decisions deterministic; keep reporting failures non-blocking where possible.
• Prioritize low-cognitive-load defaults (self-service templates over manual config work).
• Treat SARIF, run/summary JSON, and Action inputs/outputs as compatibility contracts.

Default Gate Go/No-Go Checklist (P0)¶

Zodra alle items hieronder groen zijn: "default gate ready".

Definition of "default gate ready": All ⬜ → ✅

0. Epics Overview¶

De onderstaande epics groeperen het DX-plan in uitvoerbare eenheden. Per epic: goal, priority (P0/P1), stories, acceptance criteria, effort. De gedetailleerde patchlist staat in de secties 1–8.

Epic E1: Blessed init & CI on-ramp¶

Stories:

Acceptance criteria:

• assay init --ci → .github/workflows/assay.yml bevat assay-action@v2 (golden/contract test).
• Docs: init --ci = blessed; init-ci = alias; CI-integration + example repos link.
• (P1) CI of smoke: assay run in dx-demo-node en dx-demo-python slaagt.
• (P1) assay init kan een minimale trace + suite scaffolden die lokaal direct een bruikbaar signaal geeft.

Epic E2: PR feedback UX (JUnit, SARIF, fork)¶

Stories:

Acceptance criteria:

• JUnit artifact + annotations bij failure met blessed snippet.
• Unit: elk SARIF-result heeft locations.length ≥ 1; contract: schema 2.1.0 + upload-smoke.
• (P1) Truncatie + N omitted in run summary/SARIF description.
• (P1) Docs: fork = job summary only.

Epic E3: Exit codes & reason code registry¶

Stories:

Acceptance criteria:

• Missing trace → exit 2, reason_code E_TRACE_NOT_FOUND (v2); v1 legacy beschikbaar via --exit-codes=v1.
• Judge unavailable (mock) → exit 3, reason_code E_JUDGE_UNAVAILABLE.
• summary.json bevat reason_code_version; reason_code in console, job summary, summary.json (en waar van toepassing SARIF).
• run.md en troubleshooting.md in lijn met gedrag; ADR-019 compatibility beschreven.

Epic E4: Ergonomie & debuggability¶

Stories:

Acceptance criteria:

• Config/trace/test failure → stdout bevat minstens één suggestie (assay doctor / explain / baseline).
• summary.json bevat slowest_tests (max 5), cache_hit_rate, phase_timings; console toont ze.
• Suite met 10+ tests → console toont progress (bijv. 3/10). ✅ PR #164 (JoinSet, throttle, formatter tests).

Epic E5: Observability & privacy defaults¶

Stories:

Acceptance criteria:

• Golden tests: export (OTel, replay, SARIF, job summary) met default bevat geen prompt/response body.

Epic E6: P1.3 MCP Auth Hardening (Security baseline)¶

Scope split (beheersbare delivery):

Stories:

Acceptance criteria:

• E6a DoD: resource + iss/aud; alg/typ/crit tests; JWKS stale-while-revalidate + kid-miss + max-keys; no pass-through bewezen; config gedocumenteerd.
• E6b DoD (optional): DPoP jti replay cache + htu/htm strict (when enabled via feature flag).

Epic E7: P1.1 Judge Reliability MVP¶

Stories:

Acceptance criteria:

• DoD §8.2.10: randomized order + seed (summary.json + job summary); rerun-on-instability; max extra judge calls per run; config-first policies; metrics in CI-run; multi-judge placeholder.

Epic E8: P1.2 OTel GenAI (Observability)¶

Stories:

Acceptance criteria:

• DoD §8.3.5: semconv version in config/manifest; cardinality tests; redaction golden tests; config observability.md.

Epic E9: Replay Bundle (DX + forensic)¶

Stories:

Acceptance criteria:

• DoD §8.4.7: toolchain + seeds in manifest; replay roundtrip; scrubbed policy getest; signature placeholder.

Epics: volgorde & afhankelijkheden¶

Totale effort (indicatief): P0 ~3–4 dagen, P1 DX ~2–3 dagen, P1 SOTA ~8–12 dagen (zie §8.6).

1. First 15 minutes: init as blessed on-ramp¶

1.1 Template drift (v1 → v2 action in init --ci)¶

Problem: assay init --ci (and assay init-ci --provider github) generate a workflow that uses assay-action@v1 and assay_version: "v1.4.0", while the recommended and documented action is assay-action@v2. Trust break in minute 5.

Fix: Init-generated GitHub workflow MUST use the blessed v2 template. Belangrijk: GitHub Actions ondersteunt geen semver ranges in uses: owner/repo@ref. Opties: moving major tag @v2 (aanbevolen DX-default), exact tag @v2.12.3, of pinned SHA voor supply-chain strictness.

Test cases:

• assay init --ci in empty dir → .github/workflows/assay.yml bevat exact Rul1an/assay/assay-action@v2 en geen v1-referentie (expliciete assertion op deze string in contract test).
• assay init-ci --provider github → zelfde output.
• Golden snapshot van CI_WORKFLOW_YML in tests (e.g. tests/fixtures/contract/) met assertion op action path.

1.2 One blessed entrypoint: init --ci vs init-ci¶

Problem: Two ways to do the same thing (assay init --ci vs assay init-ci) weakens "one blessed flow" (ADR-019).

Fix: Choose one as blessed; document the other as alias.

Decision (to document): Blessed = assay init --ci. assay init-ci remains as alias (no removal) to avoid breaking existing scripts.

Test cases:

• Both commands produce byte-identical .github/workflows/assay.yml when using same provider (after 1.1 is done).

1.3 One-click DX demo repos (P1)¶

Problem: No minimal Node/Python example repo that demonstrates 0 → CI gate (clone, run, PR with annotations).

Fix: Add two example directories with minimal app + 1 test + working workflow + baseline flow.

Test cases:

• CI job in this repo (or local) runs assay run in examples/dx-demo-node and examples/dx-demo-python and exits 0 (or document as manual smoke).

2. PR feedback UX¶

2.1 JUnit: default + native annotations (blessed snippet)¶

Problem: JUnit is not default in the action; no single blessed snippet for "failures as annotations" and "where is junit.xml".

Fix: Action heeft escape hatch (teams willen soms alleen SARIF of alleen job summary). Default "works", geen lock-in.

Test cases:

• Contract test: output path voor JUnit is het gekozen pad (default .assay/reports/junit.xml).
• CI workflow met blessed snippet produceert JUnit artifact en annotations bij failure (manual of e2e).

2.2 SARIF: always one location + upload contract + limits (P0/P1)¶

Problem: GitHub upload can fail with "expected at least one location". No contract test. No handling for result/size limits.

Fix:

Test cases:

• Unit: every result in generated SARIF has locations length ≥ 1.
• Contract: generated SARIF passes schema 2.1.0 and contains at least one location per result.
• Optional: CI step that uploads a minimal SARIF (1 result, 1 location) to verify upload-sarif accepts it.

2.3 Fork PR: no SARIF/comment; fallback to job summary (P1)¶

Problem: Fork PRs cannot upload SARIF or post comments (permissions). Users should get feedback only via job summary.

Fix: Job summary altijd kernresultaten bevatten, zodat devs bij beperkte permissies toch feedback zien (ook bij "expected checks" zonder artifacts).

Test cases:

• Documented behaviour; optional: trigger from fork en assert no upload/comment, summary bevat kernresultaten.

3. Exit codes: remove DX landmine (P0)¶

Problem: run.md says exit 3 = "Trace file not found"; ADR-019 wants 3 = "infra/judge unavailable". Redefining 3 breaks existing users/CI.

Fix (SOTA): Stable, machine-readable reason code registry (decoupled from exit code). Coarse exit codes 0/½/3; expliciete compat switch; reason_code in alle outputs; downstream tooling schakelt op reason_code, niet op exit code.

Test cases:

• Missing trace → exit 2, reason_code E_TRACE_NOT_FOUND (v2); met --exit-codes=v1 → legacy exit 3.
• Judge unavailable (mock) → exit 3, reason_code E_JUDGE_UNAVAILABLE.
• reason_code aanwezig in console output, summary.json (incl. reason_code_version), en waar van toepassing job summary/SARIF.
• run.md and troubleshooting.md match behaviour.

4. Ergonomie & debuggability¶

4.1 Default "next step" in every error (P1)¶

Problem: Not every exit≠0 ends with 1–2 concrete commands. Te veel next steps = noise; niemand leest het.

Fix: Context-aware next steps; max 2 per exit.

Test cases:

• Trigger config error, missing trace, failing test; stdout bevat max 2 suggesties (assay doctor / explain / baseline, context-afhankelijk).

4.2 Performance-DX: slowest 5, cache hit rate, phase timings (P1)¶

Problem: No "slowest 5 tests", "cache hit rate", or "total time per phase" in console or summary.

Fix:

Test cases:

• Run suite with multiple tests; summary.json contains slowest_tests (max 5), cache, timings; console shows them.

4.3 Progress UX: N/M tests, ETA-ish (P1)¶

Problem: Long suites have no "N/M done, ETA" feedback.

Fix:

Test cases:

• Run suite with 10+ tests; console shows progress lines (e.g. 3/10).

5. Observability: privacy-safe defaults (P1)¶

Problem: GenAI events (prompt/response capture) are not everywhere; default should not export prompt/response content. In 2026 is dit "table stakes".

Fix: Concreet waar prompts/response bodies nooit mogen staan (default):

Test cases:

• Golden tests: OTel export, replay bundle, SARIF, job summary met default config bevatten geen prompt/response body (of alleen hash/digest indien gedocumenteerd).

6. Backlog summary (copy-paste for issues)¶

Elk item is gekoppeld aan een epic (zie §0).

P0 (must-have before default gate)¶

P1 (SOTA)¶

7. File-level checklist (patchlist)¶

8. P1 SOTA Implementation (Judge, Security, Observability, Replay)¶

Status: Planned (Updated: Bleeding Edge Jan 2026) Priority Order: P1.3 → P1.1 → P1.2 → Replay Bundle Rationale: Security baseline first (hard invariant), then judge reliability (CI signal), then observability (debugging), then DX (replay). Review Score: 9.2/10 → 9.7/10 with bleeding edge additions below.

8.1 P1.3 MCP Auth Hardening (Security Baseline)¶

Goal: OAuth 2.0 Security BCP compliance + sender-constrained tokens where applicable.

8.1.1 Resource Indicators (RFC 8707)¶

8.1.2 DPoP (Sender-Constrained Tokens) — Optional Hardening¶

8.1.3 Bleeding Edge: Alg/Typ/Crit Hardening (JWT Footguns)¶

8.1.4 Bleeding Edge: Replay Defense (DPoP)¶

8.1.5 Bleeding Edge: JWKS Caching "Done Right"¶

8.1.6 Negative Test Suite¶

8.1.7 Definition of Done¶

• resource enforced + iss/aud validated conform OAuth BCP
• Alg/typ/crit confusion tests (bleeding edge)
• JWKS with stale-while-revalidate + kid-miss refresh + max-keys
• DPoP jti replay cache + htu/htm strict (when enabled)
• "No pass-through" proven in tests (logs + downstream aud)
• Config documented in docs/reference/config/mcp-server.md

Effort: 2–3 days

DX Impact: Fewer "mysterious 401/403" errors — developers understand what to fix via reason codes.

8.2 P1.1 Judge Reliability MVP (CI Signal/Noise)¶

Goal: Reduce flakiness, add bias mitigation, structured uncertainty handling.

8.2.1 Borderline Band + Adaptive Calibration¶

8.2.2 Bleeding Edge: Randomized Order as DEFAULT¶

Instead of always A/B → B/A test: randomized order (with seed) is DEFAULT in CI for pairwise comparisons.

This makes position bias visible without extra calls.

8.2.3 Order-Invariance (Bias Mitigation)¶

8.2.4 Bleeding Edge: Rerun on Instability (Not Just Borderline)¶

Rerun triggers expanded beyond borderline:

# Config example
judge:
  rerun_triggers:
    - borderline      # score in [0.4, 0.6]
    - low_margin      # |score - 0.5| < margin_threshold
    - order_flip      # A/B vs B/A disagreement
    - high_variance   # std_dev > variance_threshold

8.2.5 Rerun Strategy (2-of-3 Majority)¶

if first_run NOT in rerun_triggers:
    return verdict (done, 1 call)
elif first_run triggers rerun:
    run second
    if first == second:
        return verdict (done, 2 calls)
    else:
        run third
        return majority(first, second, third) (done, 3 calls)

8.2.6 Output Metrics¶

8.2.7 Bleeding Edge: Config-First Policies per Suite Type¶

# Config example
suites:
  - name: security_checks
    type: security
    uncertain_policy: fail_closed
  - name: quality_metrics
    type: quality
    uncertain_policy: quarantine

8.2.8 Fail Modes: Split "Uncertain" from "Unavailable"¶

8.2.9 Future: Multi-Judge Support (Placeholder)¶

# Structure for later: 2 different judge models (cheap + strong)
judge:
  models:
    - name: fast
      model: gpt-4o-mini
      role: first_pass
    - name: strong
      model: gpt-4o
      role: tiebreaker  # only on disagreement

8.2.10 Definition of Done¶

• Randomized order default with seed in summary.json + job summary
• Cost guardrails: judge.max_extra_calls_per_run (default 2); warning logged when cap reached
• Rerun-on-instability (borderline + low_margin + order_flip + high_variance)
• Config-first policies per suite type (security/quality/regression)
• CI-run produces consensus_rate, flip_rate, abstain_rate, margin
• Reason codes E_JUDGE_UNCERTAIN, E_JUDGE_UNAVAILABLE
• Multi-judge config placeholder (structure, not full implementation)
• Audit E: Robust JSON Parsing (Greedy stream seeker)
• Audit F: Audit Evidence Pack (E7-AUDIT.md)

Effort: 2–3 days (MVP), +1 day for tuning PRs

DX Impact: Fewer flaky failures → devs trust CI again. "Uncertain" with reason_code + next_step → faster debugging.

8.3 P1.2 OTel GenAI (Observability)¶

Goal: OpenTelemetry GenAI semantic conventions compliance; privacy-safe defaults.

8.3.1 Bleeding Edge: Semconv Version Gating¶

Critical: GenAI semconv evolves rapidly. Without version gating, backward compat breaks.

# Config
otel:
  genai_semconv_version: "1.28.0"  # or "latest"

8.3.2 Span Layers¶

8.3.3 Bleeding Edge: Low-Cardinality Enforcement (Hard)¶

8.3.4 Bleeding Edge: Composable Redaction Policies¶

otel:
  capture_prompts: false  # default
  redaction_policies:
    - strip_secrets      # API keys, tokens
    - strip_file_paths   # Local paths
    - strip_pii          # Email, phone (regex)
    - custom: "s/password=.*/password=REDACTED/"

8.3.5 Definition of Done¶

• Semconv version gating in config + manifest
• Low-cardinality enforcement tests (labels bounded)
• Spans conform GenAI semconv (versioned)
• Composable redaction policies
• Golden tests: default = no prompt; full = redacted content
• Config documented in docs/reference/config/observability.md

Effort: 1–2 days

DX Impact: "Why is this slow/flaky" → spans/metrics immediately available.

8.4 Replay Bundle (DX + Forensic)¶

Goal: Reproducible test runs from a single artifact; supply-chain aware.

8.4.1 Bundle Format¶

.assay/replay.bundle/
├── manifest.json          # Provenance + file digests + toolchain
├── config/
│   ├── eval.yaml
│   └── policy.yaml
├── traces/
│   └── input.jsonl
├── cassettes/             # VCR recordings (scrubbed)
│   └── openai/
│       └── *.json
├── baseline/
│   └── baseline.json
└── toolchain/             # NEW: for true reproducibility
    ├── Cargo.lock
    └── cargo-metadata.json

8.4.2 Bleeding Edge: Toolchain Capture (Critical for Reproducibility)¶

Without toolchain capture, "replay works on my machine" is common. Include:

{
  "schema_version": 2,
  "created_at": "2026-01-30T12:00:00Z",
  "assay_version": "2.12.0",
  "git_sha": "abc123...",
  "workflow_run_id": "12345678",
  "toolchain": {
    "rustc": "rustc 1.84.0 (9fc6b4312 2025-01-07)",
    "cargo": "cargo 1.84.0 (66221abde 2024-11-19)",
    "target_triple": "aarch64-apple-darwin",
    "cargo_lock_digest": "sha256:abc123...",
    "cargo_metadata_snapshot": "sha256:def456..."
  },
  "runner": {
    "os": "Linux",
    "os_version": "Ubuntu 22.04.3 LTS",
    "runner_image": "ubuntu-latest",
    "uname": "Linux 6.5.0-1025-azure x86_64"
  },
  "files": {
    "config/eval.yaml": { "sha256": "...", "size_bytes": 1234 },
    "traces/input.jsonl": { "sha256": "...", "size_bytes": 5678 }
  },
  "bundle_digest": "sha256:...",
  "tool_versions": {
    "openai_sdk": "1.x.x",
    "reqwest": "0.12.x"
  }
}

Captured files: - Cargo.lock (exact dependency versions) - cargo metadata --format-version 1 snapshot - rustc -Vv output - Runner environment metadata

8.4.3 Bleeding Edge: Deterministic Seed Logging¶

For judge reliability: seed is logged → replay with same seed = same order.

{
  "determinism": {
    "judge_order_seed": 42,
    "random_seed": 12345,
    "timestamp_frozen": false
  }
}

8.4.4 Bleeding Edge: Scrubbed Cassettes Policy¶

SOTA: Scrubbing deny-by-default (allowlist van toegestane velden, niet blocklist). Zo blijft bundle veilig bij nieuwe velden.

replay:
  include_prompts: false        # default
  scrub_cassettes: true         # remove secrets from VCR cassettes
  scrub_policy: "default"       # allowlist (niet blocklist)

8.4.5 Privacy: Minimal Secrets Risk¶

8.4.6 CLI Interface¶

# Create bundle from last run
assay bundle create --output replay.bundle

# Replay bundle (offline, VCR mode)
assay replay --bundle replay.bundle

# Replay with network (re-run against live providers)
assay replay --bundle replay.bundle --live

# Replay with specific seed (for judge order reproducibility)
assay replay --bundle replay.bundle --seed 42

8.4.7 Definition of Done¶

• Toolchain capture (rustc, cargo, lock, metadata, runner)
• Deterministic seed logging for reproducibility
• Manifest with file digests + provenance
• assay replay --bundle reproduces (VCR, deterministic seeds)
• Scrubbed cassettes policy + tests
• Privacy: no prompts/secrets unless opt-in
• Signature placeholder (structure for later Sigstore/cosign)

Effort: 2–3 days

DX Impact: Reviewers can reproduce "exactly this" locally. Bundle is often the "next step" on failures.

8.5 P1 File-Level Checklist (Updated)¶

8.6 P1 Effort Summary¶

DX-items priority: #10 (next steps) → #11 (perf DX) → #13 (privacy) — highest impact first.

8.7 PR Sequence Blueprint¶

Recommended PR structure for implementation:

PR 1: P1.3 MCP Auth Hardening
  ├── auth/resource.rs (RFC 8707)
  ├── auth/jwt_validation.rs (alg/typ/crit)
  ├── auth/jwks.rs (cache improvements)
  ├── auth/dpop.rs (optional, behind feature flag)
  └── tests/auth_negative.rs

PR 2: P1.1 Judge Reliability
  ├── judge/borderline.rs
  ├── judge/order.rs (randomized default)
  ├── judge/rerun.rs (instability triggers)
  ├── judge/policy.rs (suite-type policies)
  └── tests/judge_reliability.rs

PR 3: P1.2 OTel GenAI
  ├── otel/genai.rs (semconv versioned)
  ├── otel/metrics.rs (low-cardinality)
  ├── otel/redaction.rs (composable)
  └── tests/otel_cardinality.rs

PR 4: Replay Bundle
  ├── replay/bundle.rs
  ├── replay/manifest.rs (toolchain, seeds)
  ├── replay/scrub.rs
  └── tests/bundle_roundtrip.rs

DX Mini-PRs (parallel):
  ├── #10: suggest_next_steps()
  ├── #11: slowest 5 + phase timings
  └── #13: privacy defaults + redaction tests

9. References¶

• §0 Epics Overview — epics E1–E9 met stories, acceptance criteria en effort
• DX-REVIEW-MATERIALS.md — current DX review materials
• ADR-019 PR Gate 2026 SOTA — performance, DX, security, judge, observability
• ROADMAP — strategic roadmap
• reference/cli/run.md — run exit codes and outputs
• guides/troubleshooting.md — troubleshooting guide