Confidently wrong, or quietly compromised?

An AI system fails in two different ways. It gives an answer that sounds right and isn't, because nobody defined what "correct" means here. Or it does exactly what an attacker wanted, because nobody told it what not to trust. Different questions, different testing, and most teams have only budgeted for one of them, if either.

This is what AI evaluation is for. Not because AI is unreliable, often it is excellent. Because black-box, non-deterministic behaviour needs far more rigorous checking than code a human wrote and can explain, and most teams have not adjusted their QA to match.

For any regulated or high-stakes deployment, you need the failure rate empirically, before you go live. Not the vendor's benchmark. Yours. Tested in your environment, against your data, on your tasks.

Security asks a different question to evaluation. Not whether the system performs well, but whether it can be made to misbehave. Prompt injection and sensitive information disclosure top the current OWASP Top 10 for LLM Applications, alongside excessive agency, data and model poisoning, and improper output handling, and each is a live attack surface wherever a model reads untrusted input or acts with delegated authority.

We test by adversarial example: crafted inputs and hidden instructions in retrieved content, attempts to extract system prompts or training data, and requests that push an agent beyond its intended scope. Findings are reproducible and mapped to the OWASP category they sit under.

AI evals: the method

The method is black-box testing applied to non-deterministic systems. Define the inputs. Define what correct looks like. Run at scale. Record pass and fail. Characterise the failure patterns. The AI-specific challenge is the oracle problem, deciding what "correct" means when outputs vary, which we handle through structured output evaluation, LLM-as-judge protocols, and human review sampling.