ProofAgent-Harness 0.11.0 — Release Notes

Released 30 July 2026 · Apache-2.0 · Python 3.10+

0.11.0 is the latest release of ProofAgent-Harness, open-source infrastructure for auditable AI agent evaluation and governance. It runs multi-turn adversarial conversations against a live agent, and artifact review of finished deliverables, both through the same multi-jury auditable evaluation and scoring across six production-critical metrics, with every score linked to the evidence that justified it. The headline of this release is PAI — the ProofAgent Governance Readiness Index: one 0–100 number over the four things a deployment decision actually depends on. Does the agent behave under adversarial pressure, is its context engineered to hold, does it meet your compliance obligations, and is it governed. LLM agnostic, runs locally, and gates CI on the result.

TL;DR. pip install -U proofagent-harness. 0.11.0 adds PAI, a 0–100 readiness index over four independently measured axes (E behaviour · Q context · C compliance · G governance), with a hard block for anything dangerous and a completeness rule that refuses to issue a verdict on partial evidence. Also new: a repeatable compliance axis across a catalog of 25 frameworks, --adaptive-turns to size the run from its risk, --fresh and replay for repeatable numbers, framework-steered trap selection, and verified proof quotes in every finding. Scores are not directly comparable with earlier releases, so re-baseline before gating on a version delta. Python 3.10+.

Install

pip install -U proofagent-harness                 # latest release (0.11.0)
pip install "proofagent-harness[artifact]"        # + PDF/DOCX/HTML/image parsers for artifact mode

proof version       # → proofagent-harness 0.11.0
proof traps stats   # → 183 traps · 11 families · 40 domains · 6 metrics

New in 0.11.0: PAI — the ProofAgent Governance Readiness Index

PAI (ProofAgent Governance Readiness Index) is a 0–100 score for whether an AI agent is admissible for production, measured across four axes: behaviour, context, compliance, and governance.

A benchmark score tells you how an agent performs. A release owner needs to know whether it is admissible — whether there is enough evidence across every deployment obligation to ship it. Those are different questions, and an agent can ace the first while failing the second: accurate but non-compliant, or well-behaved but ungoverned. PAI answers the second one, as a single 0–100 number with a grade and a readiness band.

proof run agent.py \
    --context-dir ./my_agent/context \
    --governance-profile ./governance.yaml \
    --assess-context \
    --assess-compliance \
    --json report.json --markdown report.md

# score a finished report, or axes you already have
proof pai --report report.json
proof pai -E 82 -Q 60 -C 61 -G 56

# fail CI below a bar, and refuse to pass on incomplete evidence
proof pai --report report.json --min-pai 70 --require-complete

PAI is computed on every run, printed after it, carried in every report as report.pai, and gateable in CI. proof pai exits 0 when the bar is met, 1 below the bar or on incomplete evidence, and 2 when the run is hard blocked.

Why it matters

Three properties do the work, and each one exists to close a way a single number can mislead you.

• A strong axis cannot rescue a weak one. An agent that behaves impeccably but has no compliance evidence does not average its way to a passing score. Cheap and dangerous can never read well.
• Anything dangerous caps the score outright. A prohibited use case, a breach of the safety floor, or a critical finding pins the index into the F band regardless of the average — and the report names which reason capped it, so the decision is auditable rather than asserted. A governance gate saying "below this tier's bar" is surfaced as a reason but does not cap, so attaching a stricter profile never scores an agent below the same agent run ungoverned.
• Absence of evidence is not evidence of readiness. All four axes are required. A run missing any of them reports PAI-Partial with readiness indeterminate and no verdict, because incompleteness must block an approval and never manufacture one. It does not block a rejection: a dangerous run still reads blocked on partial evidence.

The report carries two figures. score is the one a release decision is made against. raw_score is the same aggregate before any cap — read it too, because once an agent is hard blocked the headline stops moving and only the uncapped figure shows whether successive versions are improving.

PAI (ProofAgent Governance Readiness Index)  70.2 +/- 7.3 / 100   C · Healthy
READY WITH CAVEATS   (PAI-Complete)

  Q  Context engineering      74%
  E  Behavioral evaluation    92%
  C  Framework compliance     54%  ← weakest
  G  Governance               66%

Full reference: PAI documentation · runnable example: examples/14_pai_readiness_index.py

Framework Compliance — the C Axis

--assess-compliance maps a run onto named controls from a catalog of 25 frameworks — EU AI Act, NIST AI RMF, ISO/IEC 42001, SOC 2, GDPR and more — and returns a per-framework coverage table where every status names the observation behind it.

proof run agent.py --assess-compliance --frameworks "EU AI Act,SOC 2"

Why it matters

"Not assessed" is neutral, never a zero. A control the run could not exercise is left out of the score rather than counted as a failure or guessed as a pass, because the difference between we checked and it is fine and we did not check has to survive into the report — that distinction is the whole value of the artifact to an auditor. New in this release, --frameworks also steers which traps run, so a framework you declare is actually exercised instead of coming back mostly not_evaluated.

Reference: compliance documentation

Context Engineering — the Q Axis

--assess-context grades the context the agent runs on — its system prompt, tool schemas, and whether you supplied grounding knowledge — across seven criteria, each with a specific gap and a token-savings estimate.

# --context-dir is the context that gets GRADED
# --domain-knowledge-dir is what the agent is expected to be grounded ON
proof run agent.py \
    --context-dir ./my_agent/context \
    --domain-knowledge-dir ./knowledge/ \
    --assess-context

Why it matters

Bad context is the upstream cause of most bad behaviour: an agent with no injection defence in its prompt will fail injection traps no matter which model is behind it. New in this release, the grade weighs the behavioural result as well as appearing in the report — a failure in an area your context leaves undefended counts for more than the same failure in an area it covers. Weak context does not create failures, it amplifies the ones the agent already had. Leave the flag off and scoring is unchanged.

Reference: context engineering documentation · runnable example: examples/12_context_engineering.py

Sizing the Run: --adaptive-turns

How many adversarial turns to run is a coverage decision. Too few and whole families of attack never get tried, and the score then reflects what you happened to test rather than how the agent behaves.

proof run agent.py --adaptive-turns     # the harness sizes it from this configuration
proof run agent.py --turns 15           # fixed; the recommendation is still reported

Why it matters

With --adaptive-turns the harness accounts for the risk tier, the frameworks you declared, the weaknesses found in the context, the tool surface, and the domains in scope. Give an explicit --turns N instead and that is what runs, but the recommendation is still computed and reported next to it, so a short run reads as a deliberate choice rather than an oversight:

! Ran 15 adversarial turn(s); the planner recommends 37 for this configuration
  (+8 high-risk tier (high); +8 for 18 exposed behaviour(s) in the context)

Reference: sizing the run

Repeatable Runs: --fresh and Replay

For a number you can put in a report and reproduce later, run the agent once and score that stored transcript as often as you like. --fresh forces a new agent run; omitting it replays a stored transcript instead.

# 1. record
AGENT_TEMPERATURE=0 proof run agent.py --seed 5 --turns 15 --fresh --json r1.json

# 2. re-score the SAME transcript (omit --fresh)
AGENT_TEMPERATURE=0 proof run agent.py --seed 5 --turns 15 --json r2.json

Why it matters

Re-running the agent and re-scoring a transcript measure two different things. Replaying a transcript tells you how stable the scoring is. Re-running the agent tells you how stable the agent is, and that is usually the larger term: measured on a real agent at temperature 0.2 with a fixed seed, one run called a barred email tool three times and the next touched only safe tools. If a fresh run swings, read it as a finding about the agent before suspecting the harness. Every metric also carries a confidence value — below roughly 0.90 the jury was divided, and that metric is the one most likely to shift on a re-score.

Reference: reproducibility documentation

Findings You Can Check

Every finding is presented as problem · proof · fix, and in 0.11.0 the proof field is a verbatim quote from the transcript or nothing at all. If a quote cannot be produced the field is left empty rather than filled with a restatement of the problem, because an unverifiable proof is worse than none in an audit artifact. Findings also carry a strengths list, so what the agent did correctly is reported under its own heading instead of being implied by the absence of complaint.

What Stays the Same

The core is unchanged: two evaluation modes (multi_turn and mode="artifact"), six canonical metrics, a multi-juror panel with a blind second round on disputed metrics, 11 artifact rubric packs, a 183-trap adversarial library across 11 families and 40 domains, and a one-command CI release gate via --upload with exit codes 0 pass · 1 review · 2 block. The harness still runs fully offline by default; only --upload contacts a network service. Separately from evaluating an agent you build, proof watch and proof session screen a coding agent you use as it works in your repo — Claude Code and Cursor read natively, anything else through the git working tree.

Verify Your Install and Run the Tests

The package ships its own test suite. The first three steps need no API key.

# 1. the package resolved, and the bundled data came with it
proof version                 # → proofagent-harness 0.11.0
proof traps stats             # → 183 traps · 11 families · 40 domains · 6 metrics

# 2. PAI arithmetic, no LLM and no network needed
proof pai -E 82 -Q 60 -C 61 -G 56

# 3. the full suite (624 tests, ~4s) — needs the dev extra
pip install "proofagent-harness[dev]"
pytest tests/ -q
ruff check src tests

# 4. a real evaluation. LLM agnostic — this one uses an open-weights model
#    served locally by LM Studio, Ollama, or vLLM
export OPENAI_BASE_URL=http://localhost:1234/v1
proof run examples/credit_agent/agent.py \
    --context-dir examples/credit_agent/context \
    --domain-knowledge-dir examples/credit_agent/domain_knowledge \
    --governance-profile examples/governance_profiles/credit_agent.yaml \
    --assess-context --assess-compliance \
    --llm openai/gemma-3-27b-it \
    --turns 8 --seed 42 \
    --json run.json --markdown run.md --no-upload

# 5. runnable examples
python examples/14_pai_readiness_index.py      # PAI end to end
python examples/12_context_engineering.py      # the Q axis
python examples/11_governance_gate.py          # the local release gate
python examples/04_artifact_eval.py            # artifact mode

Step 3 is the one to run after any local change. A green suite plus proof traps stats reporting 183 traps is the fastest confirmation that both the code and the bundled data installed correctly.

Compatibility

• Python 3.10+. Tested on 3.10, 3.11, and 3.12.
• LLM agnostic. Bring your own model as the harness LLM — hosted or open-weights served locally — and your agent can run on anything. --fallback-llm should name a model from a different family, so an input the primary cannot parse is retried by a model with different failure modes.
• All four PAI axes are needed for a verdict. --assess-context and --assess-compliance are opt-in; without them the run still works but reports PAI-Partial.
• Artifact mode is opt-in via mode="artifact". PDF, DOCX, HTML, and image parsing needs the [artifact] extra.
• Runs fully offline. --upload is purely opt-in, and the local release gate needs no account.

Frequently Asked Questions

What is PAI, the ProofAgent Governance Readiness Index?

PAI is a 0–100 score for whether an AI agent is admissible for production, rather than how well it performs on a benchmark. It combines four independently measured axes: E behavioural evaluation, Q context engineering, C framework compliance, and G governance. It is produced on every evaluation run and can gate a CI pipeline.

How is production readiness different from capability?

Capability measures how well an agent performs a task. Production readiness measures whether there is enough evidence across every deployment obligation to ship it. An agent can score highly on a capability benchmark while being non-compliant or ungoverned, so a high benchmark score is not evidence that an agent is safe to deploy.

What happens if an axis is missing?

All four axes are required for a verdict. A run missing any of them reports PAI-Partial with readiness indeterminate and issues no score, because absence of evidence is not evidence of readiness. Incompleteness blocks an approval but never a rejection.

Which compliance frameworks are covered?

A catalog of 25 frameworks, including the EU AI Act, NIST AI RMF, ISO/IEC 42001, SOC 2 and GDPR. Each control returns met, partial, attention, or not_evaluated.

Which LLM does it require?

ProofAgent-Harness is LLM agnostic. Bring your own model as the harness LLM, hosted or an open-weights model such as Gemma or Llama served locally through LM Studio, Ollama, or vLLM. The agent under evaluation can run on anything.

Does it send my prompts or code to a server?

No. It runs fully offline by default, including the release gate, which needs no account. Only the opt-in --upload flag contacts a network service.

Can it monitor coding agents like Claude Code and Cursor?

Yes. proof watch and proof session screen a coding agent you use as it works in your repository. Claude Code and Cursor are read natively; any other coding agent is covered through the git working tree with no plugin required.

References

• PyPI: proofagent-harness on PyPI
• GitHub release: ProofAgent-Harness v0.11.0 on GitHub
• Documentation: ProofAgent-Harness documentation · PAI · Compliance · Context engineering · Sizing the run · Reproducibility
• Research paper: ProofAgent Harness: Open Infrastructure for Adversarial Evaluation of AI Agents (arXiv:2605.24134)