02 · AI System Testing
AI coding builds faster. What has it built?
AI-assisted development is real, and it is a wonder. Code that used to take weeks now takes days. That is the productivity miracle everyone is talking about. But speed in the build phase does not mean speed everywhere. A pipeline only moves as fast as its slowest stage, and building was never the hard part. Specifying what "correct" really means, and then checking that the system actually does that and only that, always was. AI raises the stakes on both.
So the work does not disappear. It shifts. Less time spent building, more time spent specifying what good looks like and proving the system delivers it. Every study on AI-assisted coding shows the same shape. Gains in raw output, and a growing tail of review, QA and rework behind it.
This is what AI system testing is for. Not because AI is unreliable, often it is excellent. Because black-box, non-deterministic behaviour needs far more rigorous checking than code a human wrote and can explain, and most teams have not adjusted their QA to match.
For any regulated or high-stakes deployment, you need the failure rate empirically, before you go live. Not the vendor's benchmark. Yours. Tested in your environment, against your data, on your tasks.
The method
The method is black-box testing applied to non-deterministic systems. Define the inputs. Define what correct looks like. Run at scale. Record pass and fail. Characterise the failure patterns. The AI-specific challenge is the oracle problem, deciding what "correct" means when outputs vary, which we handle through structured output evaluation, LLM-as-judge protocols, and human review sampling.
| Task category | Runs | Pass | Fail | Rate |
|---|---|---|---|---|
| Contract clause extraction | 500 | 471 | 29 | 94.2% |
| Risk flag classification | 500 | 443 | 57 | 88.6% |
| Multi-step reasoning | 200 | 154 | 46 | 77.0% |
| Numerical extraction | 500 | 489 | 11 | 97.8% |
| Edge case handling | 300 | 201 | 99 | 67.0% |
Illustrative figures, not client data.
If you operate under the EU AI Act, fully applicable from August 2026, or DORA, the evidence question is no longer optional. Both ask you to show how your systems behave under test, not how the vendor's benchmark behaved. This work produces that evidence.
You get: failure rates by task category with confidence intervals, characterised failure modes (hallucination, silent omission, misclassification, compounding), and recommendations for guardrails, monitoring and human-in-the-loop design. Written for your CISO or Head of Risk, not just your engineers.
Typical engagement: two to four weeks.