Entry Points¶

This document catalogs all ways to interact with Assay: CLI commands, Python SDK methods, MCP server endpoints, and configuration files.

CLI Commands¶

All CLI commands are defined in crates/assay-cli/src/cli/args.rs and dispatched in crates/assay-cli/src/cli/commands/mod.rs.

Core Commands¶

`assay run`¶

Purpose: Execute test suite against traces Entry: crates/assay-cli/src/cli/commands/mod.rs::cmd_run() Flow: load_config() → build_runner() → Runner::run_suite() → report; writes run.json, summary.json (seeds, judge_metrics, reason_code per SPEC-PR-Gate-Outputs-v1), and console footer (Seeds line + judge metrics). See Run Output.

Key Options: - --config <PATH>: Config file (default: eval.yaml) - --trace-file <PATH>: Trace file to use - --baseline <PATH>: Baseline file for regression testing - --export-baseline <PATH>: Export baseline after run - --strict: Fail on any violation - --rerun-failures <N>: Retry failed tests N times

`assay validate`¶

Purpose: Stateless validation of traces against policy Entry: crates/assay-cli/src/cli/commands/validate.rs::run() Flow: load_config() → validate::validate() → report

Key Options: - --config <PATH>: Policy config file - --trace-file <PATH>: Trace file to validate - --format <FORMAT>: Output format (text, json, sarif)

`assay init`¶

Purpose: Initialize new Assay project Entry: crates/assay-cli/src/cli/commands/init.rs::run() Flow: Detect project type → generate eval.yaml + policy.yaml

Key Options: - --ci [github|gitlab]: Generate CI scaffolding - --gitignore: Generate .gitignore for Assay artifacts - --preset <default|hardened|dev>: Select starter policy preset (backward alias: --pack) - --list-presets: List available presets (backward alias: --list-packs) - --from-trace <PATH>: Generate config/policy from an existing trace - --heuristics: Enable heuristics for --from-trace

Trace Management¶

`assay import`¶

Purpose: Import traces from external formats Entry: crates/assay-cli/src/cli/commands/import.rs::cmd_import() Flow: Parse input format → convert to JSONL → optionally generate config

Supported Formats: - inspector: MCP Inspector session logs - jsonl: Direct JSONL import - otel: OpenTelemetry traces

Key Options: - --format <FORMAT>: Input format - --init: Auto-generate config - --out-trace <PATH>: Output trace file

`assay trace`¶

Purpose: Trace utilities (ingest, verify, precompute, MCP import) Entry: crates/assay-cli/src/cli/commands/trace.rs::cmd_trace() Flow: Execute selected subcommand (ingest, ingest-otel, verify, precompute-*, import-mcp)

`assay replay`¶

Purpose: Interactive trace replay Entry: crates/assay-cli/src/cli/commands/replay.rs Flow: Load trace → step through → inspect results Note: Replay bundle core (manifest, container writer, toolchain) is in crates/assay-core/src/replay/.

Policy Management¶

`assay generate`¶

Purpose: Generate policy from traces (learning mode) Entry: crates/assay-cli/src/cli/commands/generate.rs::run() Flow: Analyze traces → generate policy constraints → write policy.yaml

Key Options: - --from-profile <PATH>: Generate from profile - --from-trace <PATH>: Generate from trace file - --output <PATH>: Output policy file

`assay record`¶

Purpose: Capture and generate in one flow Entry: crates/assay-cli/src/cli/commands/record.rs::run() Flow: Capture traces → generate policy → save both

`assay migrate`¶

Purpose: Migrate config from old to new format Entry: crates/assay-cli/src/cli/commands/migrate.rs::cmd_migrate() Flow: Parse old config → transform → write new config

Key Options: - --config <PATH>: Config to migrate - --dry-run: Preview changes without writing

Analysis & Debugging¶

`assay doctor`¶

Purpose: Diagnose common issues Entry: crates/assay-cli/src/cli/commands/doctor.rs::run() Flow: Analyze config + traces → report issues → suggest fixes

`assay explain`¶

Purpose: Explain policy violations Entry: crates/assay-cli/src/cli/commands/explain.rs::run() Flow: Load trace → find violations → generate human-readable explanation

`assay coverage`¶

Purpose: Analyze policy coverage Entry: crates/assay-cli/src/cli/commands/coverage.rs::cmd_coverage() Flow: Load traces + policy → calculate coverage → report

Key Options: - --min-coverage <PERCENT>: Minimum coverage threshold - --trace-file <PATH>: Trace file to analyze

Baseline Management¶

`assay baseline`¶

Purpose: Manage baselines for regression testing Entry: crates/assay-cli/src/cli/commands/baseline.rs

Subcommands: - record: Record baseline from current run - check: Check against baseline - report: Show baseline report

CI Integration¶

`assay ci`¶

Purpose: CI-optimized test execution Entry: crates/assay-cli/src/cli/commands/mod.rs::cmd_ci() Flow: Similar to run but optimized for CI (strict mode, SARIF output)

`assay init-ci`¶

Purpose: Generate CI workflow files Entry: crates/assay-cli/src/cli/commands/init_ci.rs::cmd_init_ci() Flow: Generate GitHub Actions / GitLab CI config

Runtime Security¶

`assay-mcp-server` (separate binary)¶

Purpose: Start Assay MCP server/proxy Entry: crates/assay-mcp-server/src/main.rs (separate binary) Flow: Load policies → start JSON-RPC server → proxy tool calls

Key Options: - --policy-root <PATH>: Policy root directory (default: policies)

`assay monitor`¶

Purpose: Runtime eBPF monitoring (Linux only) Entry: crates/assay-cli/src/cli/commands/monitor.rs::run() Flow: Load policy → compile Tier 1 rules → load eBPF → monitor process

Key Options: - --policy <PATH>: Policy file - --pid <PID>: Process ID to monitor - --cgroup <PATH>: Cgroup to monitor

`assay sandbox`¶

Purpose: Secure execution sandbox Entry: crates/assay-cli/src/cli/commands/sandbox.rs::run() Flow: Load policy → apply Landlock → execute command

MCP Management¶

`assay mcp wrap`¶

Purpose: Wrap MCP server with policy enforcement and audit logging Entry: crates/assay-cli/src/cli/commands/mcp.rs::cmd_wrap() Flow: Load policy → spawn wrapped command → proxy tool calls → emit CloudEvents

Key Options: - --policy <PATH>: Policy file (default: assay.yaml) - --dry-run: Log decisions but do not block - --verbose: Print decisions to stderr - --label <LABEL>: Unique label for this server (tool identity) - --audit-log <PATH>: NDJSON log for mandate lifecycle events (mandate.used, mandate.revoked) - --decision-log <PATH>: NDJSON log for tool.decision events - --event-source <URI>: CloudEvents source URI (e.g. assay://org/app), required when logging enabled - -- <command> [args...]: Wrapped MCP server/process command (required)

New in v2.10: --decision-log, --event-source, --audit-log flags for mandate runtime enforcement.

`assay discover`¶

Purpose: Discover MCP servers on machine Entry: crates/assay-cli/src/cli/commands/discover.rs::run() Flow: Scan for MCP processes → list servers

`assay kill`¶

Purpose: Kill/terminate MCP servers Entry: crates/assay-cli/src/cli/commands/kill.rs::run() Flow: Find MCP processes → terminate

Advanced Features¶

`assay quarantine`¶

Purpose: Manage flaky test quarantine Entry: crates/assay-cli/src/cli/commands/mod.rs::cmd_quarantine() Flow: Mark/unmark tests as quarantined

`assay calibrate`¶

Purpose: Calibrate metric thresholds Entry: crates/assay-cli/src/cli/commands/calibrate.rs::cmd_calibrate() Flow: Analyze historical results → suggest thresholds

`assay profile`¶

Purpose: Manage multi-run profiles Entry: crates/assay-cli/src/cli/commands/profile.rs::run() Flow: Collect profiles → analyze stability

`assay evidence`¶

Purpose: Evidence management (audit/compliance) Entry: crates/assay-cli/src/cli/commands/evidence/mod.rs::run() Flow: Export/verify/lint/diff/push/pull evidence artifacts

Subcommands: - export: Export evidence bundle from Profile - verify: Verify bundle integrity and provenance - show: Inspect bundle contents (verify + table view) - lint: Lint bundle for quality and security issues (SARIF output) - diff: Compare two bundles and report changes - explore: Interactive TUI explorer (requires tui feature) - push: Upload bundle to BYOS storage (S3/Azure/GCS/local) - pull: Download bundle from BYOS storage - list: List bundles in BYOS storage

Key Options: - export --profile <PATH>: Input Profile trace - export --out <PATH>: Output bundle path (.tar.gz) - export --detail <LEVEL>: Detail level (summary, observed, full) - verify <BUNDLE>: Verify bundle (or - for stdin) - show --no-verify: Skip verification (show even if corrupt) - lint --format sarif: Output in SARIF format - lint --fail-on <SEVERITY>: Fail on severity threshold - diff <BUNDLE1> <BUNDLE2>: Compare two bundles - push <BUNDLE> --store <URI>: Upload to storage - pull --bundle-id <ID> --store <URI>: Download from storage - list --store <URI>: List bundles

`assay tool`¶

Purpose: Tool signing and verification Entry: crates/assay-cli/src/cli/commands/tool/mod.rs::cmd_tool() Flow: Generate keys, sign/verify tool definitions

Subcommands: - keygen: Generate ed25519 keypair (PKCS#8/SPKI PEM) - sign: Sign tool definition with x-assay-sig field - verify: Verify tool signature with trust policy

Key Options: - keygen --out <DIR>: Output directory for keypair - keygen --force: Overwrite existing files - sign <TOOL> --key <PRIVATE_KEY> --out <OUTPUT>: Sign tool - sign --in-place: Modify file in place (dangerous) - sign --embed-pubkey: Include public key in signature (dev only) - verify <TOOL> --pubkey <PUBLIC_KEY>: Verify with public key - verify <TOOL> --trust-policy <YAML>: Verify with trust policy - verify --allow-embedded-key: Use embedded key (dev only, insecure) - verify --quiet: Only exit code, no output

Exit Codes: - 0: Verification successful - 2: Tool is unsigned (no x-assay-sig field) - 3: Key not trusted (policy violation) - 4: Signature invalid (tamper/wrong key)

`assay sim`¶

Purpose: Attack simulation (hardening/compliance) Entry: crates/assay-cli/src/cli/commands/sim.rs::run() Flow: Run attack suite → report blocked/bypassed

`assay demo`¶

Purpose: Generate demo environments with sample configs Entry: crates/assay-cli/src/cli/commands/demo.rs::run() Flow: Create sample project with traces, policies, and configs

`assay fix`¶

Purpose: Agentic policy fixing based on violations Entry: crates/assay-cli/src/cli/commands/fix.rs::run() Flow: Analyze violations → suggest/apply policy fixes

`assay setup`¶

Purpose: Interactive installer and environment setup Entry: crates/assay-cli/src/cli/commands/setup.rs::run() Flow: Interactive setup wizard

Utility Commands¶

`assay version`¶

Purpose: Show version Entry: crates/assay-cli/src/cli/commands/mod.rs::dispatch() Flow: Print version string

`assay policy`¶

Purpose: Policy management commands Entry: crates/assay-cli/src/cli/commands/policy.rs::run() Flow: Various policy operations

Python SDK Entry Points¶

Located in assay-python-sdk/python/assay/.

`AssayClient` (`client.py`)¶

Purpose: Record traces to JSONL files

Key Methods:

class AssayClient:
    def __init__(self, trace_file: str)
    def record_trace(self, trace: dict) -> None

Usage:

from assay import AssayClient

client = AssayClient("traces.jsonl")
client.record_trace({
    "tool": "filesystem_read",
    "args": {"path": "/tmp/file.txt"}
})

`Coverage` (`coverage.py`)¶

Purpose: Analyze policy coverage for traces

Key Methods:

class Coverage:
    @staticmethod
    def analyze(traces: list, min_coverage: float = 80.0) -> CoverageReport

Usage:

from assay import Coverage

coverage = Coverage.analyze(traces, min_coverage=80.0)
if not coverage.passed:
    print(f"Coverage: {coverage.score}%")

`Explainer` (`explain.py`)¶

Purpose: Explain policy violations

Key Methods:

class Explainer:
    def __init__(self, policy_file: str)
    def explain(self, trace: list) -> str

Usage:

from assay import Explainer

explainer = Explainer("policy.yaml")
explanation = explainer.explain(trace)
print(explanation)

`validate()` (`init.py`)¶

Purpose: Stateless validation function

Signature:

def validate(policy_file: str, traces: list) -> dict

Usage:

from assay import validate

result = validate("policy.yaml", traces)
assert result["passed"]

Pytest Plugin (`pytest_plugin.py`)¶

Purpose: Pytest integration for automatic trace capture

Fixtures:

@pytest.fixture
def assay_client() -> AssayClient

Markers:

@pytest.mark.assay(trace_file="traces.jsonl")
def test_agent():
    pass

GitHub Action¶

Repository: https://github.com/Rul1an/assay/tree/main/assay-action

Basic Usage¶

- uses: Rul1an/assay/assay-action@v2

With Options¶

- uses: Rul1an/assay/assay-action@v2
  with:
    bundles: '.assay/evidence/*.tar.gz'
    fail_on: error
    sarif: true
    comment_diff: true

Inputs¶

Input	Default	Description
`bundles`	Auto-detect	Glob pattern for evidence bundles
`fail_on`	`error`	Fail threshold: `error`, `warn`, `info`, `none`
`sarif`	`true`	Upload to GitHub Security tab
`comment_diff`	`true`	Post PR comment (only if findings)
`baseline_key`	-	Key for baseline comparison
`write_baseline`	`false`	Save baseline (main branch only)

Outputs¶

Output	Description
`verified`	`true` if all bundles verified
`findings_error`	Count of error-level findings
`findings_warn`	Count of warning-level findings
`reports_dir`	Path to reports directory

Permissions Required¶

permissions:
  contents: read
  security-events: write
  pull-requests: write

MCP Server Endpoints¶

The MCP server (assay-mcp-server) exposes tools via JSON-RPC over stdio.

Tool: `assay_check_args`¶

Purpose: Validate tool arguments before execution

Request:

{
  "tool": "assay_check_args",
  "arguments": {
    "target_tool": "apply_discount",
    "args": { "percent": 50 }
  }
}

Response (violation):

{
  "allowed": false,
  "violations": [
    {
      "field": "percent",
      "value": 50,
      "constraint": "max: 30",
      "message": "Value exceeds maximum"
    }
  ]
}

Response (valid):

{
  "allowed": true,
  "violations": []
}

Tool: `assay_check_sequence`¶

Purpose: Validate tool call sequence

Request:

{
  "tool": "assay_check_sequence",
  "arguments": {
    "candidate_tool": "delete_customer",
    "previous_calls": ["get_customer"]
  }
}

Response: Similar structure to assay_check_args

Tool: `assay_policy_decide`¶

Purpose: General policy decision endpoint

Request: Tool call with arguments

Response: Allow/deny decision with violations

Configuration Files¶

`eval.yaml`¶

Purpose: Main evaluation configuration Location: Project root (default) Schema: Defined in assay-core::config

Key Sections: - version: Config version - suite: Suite name - model: LLM model configuration - tests: Test cases - settings: Execution settings

`policy.yaml`¶

Purpose: Policy constraints Location: Specified in eval.yaml or default policy.yaml Schema: Defined in assay-core::policy_engine

Key Sections: - tools: Tool-specific constraints - sequences: Sequence rules - blocklist: Blocked tools/patterns

Trace Files (`.jsonl`)¶

Purpose: Recorded agent behavior Format: JSON Lines (one JSON object per line) Schema: Defined in assay-core::trace::schema

Example:

{"tool": "filesystem_read", "args": {"path": "/tmp/file.txt"}}
{"tool": "http_request", "args": {"url": "https://api.example.com"}}

Environment Variables¶

`RUST_LOG`¶

Purpose: Control logging level Values: debug, info, warn, error Default: info

`MCP_CONFIG_LEGACY`¶

Purpose: Enable legacy config mode Values: 1 to enable Default: Disabled

`ASSAY_STRICT_DEPRECATIONS`¶

Purpose: Fail on deprecated features Values: 1 to enable Default: Disabled

Exit Codes & Reason Codes¶

Exit Codes (Coarse, CI-Compatible)¶

Code	Name	When Used
0	`EXIT_SUCCESS`	All tests pass
1	`EXIT_TEST_FAILURE`	One or more tests fail, policy violation
2	`EXIT_CONFIG_ERROR`	Invalid configuration, missing files, parse errors
3	`EXIT_INFRA_ERROR`	Judge unavailable, rate limit, timeout, network error
4	`EXIT_WOULD_BLOCK`	Sandbox/policy would block execution

Reason Codes (Fine-Grained, Machine-Readable)¶

Reason codes provide precise error identification for automation and debugging.

Config/User Errors (Exit 2)¶

Code	Description	Next Step
`E_CFG_PARSE`	Config file parse error	`assay doctor --config <file>`
`E_TRACE_NOT_FOUND`	Trace file not found	Check path exists
`E_MISSING_CONFIG`	Required config missing	`assay init`
`E_BASELINE_INVALID`	Baseline file invalid	`assay baseline record`
`E_POLICY_PARSE`	Policy syntax error	`assay policy validate <file>`
`E_INVALID_ARGS`	Invalid CLI arguments	`assay --help`

Infrastructure Errors (Exit 3)¶

Code	Description	Next Step
`E_JUDGE_UNAVAILABLE`	Judge/LLM service unavailable	Check API key, retry
`E_RATE_LIMIT`	Rate limit hit	Wait, reduce concurrency
`E_PROVIDER_5XX`	Provider returned 5xx	Retry, check status page
`E_TIMEOUT`	Request timeout	Increase timeout, check network
`E_NETWORK_ERROR`	Network connection failed	Check connectivity

Test Failures (Exit 1)¶

Code	Description	Next Step
`E_TEST_FAILED`	Test assertion failed	`assay explain <test-id>`
`E_POLICY_VIOLATION`	Policy rule violated	Review policy or fix agent
`E_SEQUENCE_VIOLATION`	Wrong tool call order	Check sequence rules
`E_ARG_SCHEMA`	Argument schema invalid	Check tool argument schema
`E_JUDGE_UNCERTAIN`	Judge returned uncertain	Review borderline result

Exit Code Compatibility¶

# Use v2 exit codes (default)
assay run --exit-codes=v2

# Use v1 legacy codes (trace not found = exit 3)
assay run --exit-codes=v1

# Environment variable
ASSAY_EXIT_CODES=v1 assay run

Output Locations¶

Reason codes appear in: - Console: Last lines of output - summary.json: reason_code and reason_code_version fields - Job Summary: When running in GitHub Actions - SARIF: In ruleId / helpUri where applicable

User Flows - How these entry points are used in workflows
Codebase Overview - Implementation details
Interdependencies - How components connect
Quick Reference - Command cheat sheet
Decision Trees - When to use which command

Entry Points¶

CLI Commands¶

Core Commands¶

assay run¶

assay validate¶

assay init¶

Trace Management¶

assay import¶

assay trace¶

assay replay¶

Policy Management¶

assay generate¶

assay record¶

assay migrate¶

Analysis & Debugging¶

assay doctor¶

assay explain¶

assay coverage¶

Baseline Management¶

assay baseline¶

CI Integration¶

assay ci¶

assay init-ci¶

Runtime Security¶

assay-mcp-server (separate binary)¶

assay monitor¶

assay sandbox¶

MCP Management¶

assay mcp wrap¶

assay discover¶

assay kill¶

Advanced Features¶

assay quarantine¶

assay calibrate¶

assay profile¶

assay evidence¶

assay tool¶

assay sim¶

assay demo¶

assay fix¶

assay setup¶

Utility Commands¶

assay version¶

assay policy¶

Python SDK Entry Points¶

AssayClient (client.py)¶

Coverage (coverage.py)¶

Explainer (explain.py)¶

validate() (__init__.py)¶

Pytest Plugin (pytest_plugin.py)¶

GitHub Action¶

Basic Usage¶

With Options¶

Inputs¶

Outputs¶

Permissions Required¶

MCP Server Endpoints¶

Tool: assay_check_args¶

Tool: assay_check_sequence¶

Tool: assay_policy_decide¶

Configuration Files¶

eval.yaml¶

policy.yaml¶

Trace Files (.jsonl)¶

Environment Variables¶

RUST_LOG¶

MCP_CONFIG_LEGACY¶

ASSAY_STRICT_DEPRECATIONS¶

Exit Codes & Reason Codes¶

Exit Codes (Coarse, CI-Compatible)¶

Reason Codes (Fine-Grained, Machine-Readable)¶

Config/User Errors (Exit 2)¶

Infrastructure Errors (Exit 3)¶

Test Failures (Exit 1)¶

Exit Code Compatibility¶

Output Locations¶

Related Documentation¶

`assay run`¶

`assay validate`¶

`assay init`¶

`assay import`¶

`assay trace`¶

`assay replay`¶

`assay generate`¶

`assay record`¶

`assay migrate`¶

`assay doctor`¶

`assay explain`¶

`assay coverage`¶

`assay baseline`¶

`assay ci`¶

`assay init-ci`¶

`assay-mcp-server` (separate binary)¶

`assay monitor`¶

`assay sandbox`¶

`assay mcp wrap`¶

`assay discover`¶

`assay kill`¶

`assay quarantine`¶

`assay calibrate`¶

`assay profile`¶

`assay evidence`¶

`assay tool`¶

`assay sim`¶

`assay demo`¶

`assay fix`¶

`assay setup`¶

`assay version`¶

`assay policy`¶

`AssayClient` (`client.py`)¶

`Coverage` (`coverage.py`)¶

`Explainer` (`explain.py`)¶

`validate()` (`init.py`)¶

Pytest Plugin (`pytest_plugin.py`)¶

Tool: `assay_check_args`¶

Tool: `assay_check_sequence`¶

Tool: `assay_policy_decide`¶

`eval.yaml`¶

`policy.yaml`¶

Trace Files (`.jsonl`)¶

`RUST_LOG`¶

`MCP_CONFIG_LEGACY`¶

`ASSAY_STRICT_DEPRECATIONS`¶