Empirical, domain-specific testing of AI systems. Not benchmark scores. Quantified failure rates in your environment, against your criteria.
AI systems are probabilistic. There is no architectural path to guaranteed correct outputs. A model that scores well on a vendor benchmark may fail unpredictably on your data, your edge cases, your domain-specific tasks.
In agentic workflows, this compounds. Each step introduces uncertainty. An error in step two propagates through steps three, four, and five. By the time a human sees the output, the failure may be invisible.
For any regulated, critical, or high-stakes deployment, you need to know the failure rate empirically before you go live. Not the vendor's number. Yours.
Before you deploy AI in a regulated process, you need to know its failure rate in your environment, not the vendor's benchmark.
This is black-box testing applied to non-deterministic systems. The methodology is the same discipline Testclub has applied for twenty years. The target is new.
Define the inputs. Define what correct looks like. Run at scale. Record pass/fail. Characterise the failure patterns. The AI-specific challenge is the oracle problem: determining "correct" for non-deterministic outputs. We handle this through structured output evaluation, LLM-as-judge protocols, and human review sampling.
| Task category | Runs | Pass | Fail | Rate |
|---|---|---|---|---|
| Contract clause extraction | 500 | 471 | 29 | 94.2% |
| Risk flag classification | 500 | 443 | 57 | 88.6% |
| Multi-step reasoning | 200 | 154 | 46 | 77.0% |
| Numerical extraction | 500 | 489 | 11 | 97.8% |
| Edge case handling | 300 | 201 | 99 | 67.0% |
A quantified, characterised assessment of how your AI system performs against your acceptance criteria, tested at scale in conditions that match production.
Failure rates broken down by task category. Characterised failure modes: does it hallucinate, does it silently omit, does it misclassify, does it compound errors across steps? Confidence intervals. Recommendations for guardrails, monitoring, and human-in-the-loop design.
The output is designed for risk and compliance buyers, not just technical teams. Something your CISO or Head of Risk can act on.
Ship AI with confidence. We’ll help you understand what it can and can’t do.