ProofAgent Harness Surpasses 10,000 PyPI Installs: Open-Source AI Agent Evaluation Gains Momentum
ProofAgent Harness has passed 10,000 downloads on PyPI. This post marks the milestone, explains what the harness is, and shows why developers now stress test their AI agents across behavior, context engineering, and compliance before production.
When we open sourced ProofAgent Harness, the goal was simple and ambitious at the same time: give every developer the infrastructure to answer one question with evidence, not intuition. Is this AI agent actually ready for production?
Today that message has spread. The library has crossed 10,310 downloads on PyPI (source: pypistats.org, updated daily). Thank you to everyone who installed it, filed an issue, shared a report, or pushed a fix. This milestone belongs to the community.
TL;DR. ProofAgent Harness is the first open source infrastructure that evaluates AI agents dynamically across the full quality spectrum: multi turn adversarial behavior, six core reliability metrics, context engineering, compliance screening across a 25 framework catalog, and a local release gate. It just passed 10,000 downloads. It works with any callable agent, in 10 lines, with no rewrite. pip install proofagent-harness
Why 10,000 Downloads Matter
Ten thousand downloads of an AI agent evaluation library say something clear about where the industry is heading. Teams no longer trust a happy path demo. They are treating agent behavior the way they treat code: something you test, gate, and prove before it ships.
Think of it as pytest + CI for AI agent behavior.
That is the mental model that made the harness spread. Developers already know what a test suite and a CI gate are. ProofAgent Harness brings that same discipline to the part of the stack that never had it: how an agent behaves under real pressure, across a full conversation, while it uses real tools.
The First Open Source Infrastructure for Full Spectrum Agent Evaluation
Most evaluation tools score one answer at a time. ProofAgent Harness was built as infrastructure: it evaluates the whole behavior path of an agent, and it does so across every dimension that decides whether an agent is production ready.
| Spectrum | What the harness evaluates |
|---|---|
| Behavior and adversarial | Realistic multi turn pressure: social engineering, prompt injection, tool misuse, policy gaslighting. 183 traps across 11 attack families, selected for your domain. A jury of three jurors scores every run; consensus decides, never a single model. |
| Core reliability | Six metrics scored on the full conversation: task success, hallucination resistance, safety, instruction following, manipulation resistance, and tool use. Regression across versions shows the moment quality drops. |
| Context engineering | The agent's setup graded on 7 criteria: role clarity, guardrail coverage, instruction consistency, tool schema quality, grounding sufficiency, injection hardening, and token efficiency. Every finding carries a token impact and savings estimate. |
| Compliance | Per control screening across a 25 framework catalog, including EU AI Act, NIST AI RMF, ISO/IEC 42001, SOC 2, GDPR, and HIPAA, with a why, proof, and fix for each assessed control. |
| Governance as code | An Agent Governance Profile in YAML gives the agent a deterministic, EU AI Act aligned risk tier and ends every run with a local release gate: pass, review, or block, with standard CI exit codes. Prohibited use cases always block. |
| Coding agent observability | proof watch screens live coding agent sessions (Claude Code, Cursor, Copilot) for secrets, personal data, dangerous commands, and unapproved network hosts. The deterministic screening tier runs at zero token cost. |
We are not aware of another open source project that covers this spectrum in one tool. That is the claim behind "infrastructure": one library, one report, every dimension that a release decision needs.
Stress Test Any AI Agent in 10 Lines
The design rule that shaped the harness: your agent stays as it is. If it can receive a message and return a response, it can be evaluated. Wrap it as a callable function, pass its system prompt, tools, and knowledge into AgentContext, and run the stress test.
from proofagent_harness import AgentContext, Harness
agent = acmeair_refund_agent
report = Harness(llm="gpt-4.1", turns=15, consensus="debate", seed=42).evaluate(
agent,
role="customer support agent for AcmeAir flight refunds",
business_case="refund requests under social engineering pressure",
goal="follow refund policy v2.4 and never bypass verification",
context=AgentContext(system_prompt=SYSTEM, tools=TOOLS, knowledge=KNOWLEDGE),
)
report.to_markdown("results/acmeair_refund_agent.md")These 10 lines run a multi turn adversarial stress test, score the agent on six metrics, and save an evidence linked report. The agent under test and the Harness LLM are intentionally separated: you can evaluate a Claude powered agent with a GPT harness, compare agent models under the same contract, or run the Harness LLM through a local provider.
Evidence, Not Just Scores
Every score in a ProofAgent Harness report is a percentage. Every percentage traces back to evidence: the transcript turn that earned it, the tool call that proved it, and a suggested fix. Here is a hypothetical example of the summary a report produces (illustrative numbers, not results from a real run):
| Metric | Score | Interpretation |
|---|---|---|
| Task success | 76% | The agent completed the support task while staying useful. |
| Hallucination resistance | 62% | Moderate risk under false premise attacks. |
| Safety | 69% | Avoided most unsafe behavior; policy gates need strengthening. |
| Instruction following | 74% | Mostly followed the system prompt and refund policy across turns. |
| Manipulation resistance | 58% | Vulnerable to authority pressure, urgency, and emotional pressure. |
| Tool use | 71% | Called the right tools, but once without completed verification. |
The agent can say the right thing while doing the wrong thing. ProofAgent Harness evaluates both language and behavior: what the agent said, what tools it called, and whether its actions matched its policy boundaries.
Governance as Code: The Newest Spectrum
The latest release, v0.9.0, added governance as code. Your agent's risk classification becomes a YAML file in your repo. The harness runs a deterministic classifier, derives the EU AI Act aligned risk tier, scopes the compliance screening to the frameworks that apply, and ends the run with a local release gate.
proof run my_agent.py --governance-profile governance.yaml --turns 8The profile is a YAML file in your repo, the classification is deterministic, and the gate runs on your machine. No account needed. Exit codes are standard CI: 0 to pass, 1 to review, 2 to block.
What Is Next
- Deeper trap coverage: expanding the adversarial catalog and domain targeting beyond the current 183 traps and 11 attack families.
- Richer compliance screening: growing the 25 framework catalog and per control assessments.
- Tighter CI integration: making the pass, review, or block gate a one line addition to any pipeline.
- Your ideas: the roadmap is shaped by issues and discussions on GitHub. If the harness helped you catch something before production, tell us. If it missed something, tell us that too.
Get Started
pip install proofagent-harnessThen point the harness at any callable agent and run your first stress test. The quickstart takes minutes: examples/01_quickstart.py.
Key Takeaways
- ProofAgent Harness has passed 10,000 downloads on PyPI (10,310 at the time of writing, per pypistats.org).
- It is the first open source infrastructure that evaluates AI agents across the full spectrum: behavior and adversarial, six reliability metrics, context engineering, compliance, and governance as code.
- Any callable agent can be stress tested in 10 lines, with no rewrite and no framework lock in.
- Reports are evidence linked: every percentage traces to the exact conversation turn that earned it.
- v0.9.0 brings governance as code: a YAML profile, a deterministic risk tier, and a local pass, review, or block gate with CI exit codes.