The Evaluation Pyramid is the AI equivalent of the software testing pyramid. It acknowledges that evaluating LLMs is inherently probabilistic and requires layers of defense.
Layers of the Pyramid
- Unit Evals (Base): Fast, deterministic assertions (e.g., "Does the JSON parse?", "Are there PII leaks?"). Run on every commit.
- Model-Graded Evals (Middle): Using stronger models to evaluate the output of weaker/faster models against a rubric. Run on nightly builds.
- Human Evals (Top): Expensive, slow, but highly accurate manual review by domain experts. Used sparingly for golden datasets.
Without a structured Evaluation Pyramid, engineering teams are flying blind and cannot confidently deploy new prompts or models.
