VERITAS: measuring frontier models on tests they never saw

VERITAS is a live evaluation harness that ranks frontier AI models on how well they write code — measured the only way that stays honest over time: against a suite of hidden tests the models never had a chance to memorize. It presents the results as a ranked leaderboard, a coverage matrix showing where each model is strong or weak, and a task explorer for drilling into individual problems.

The problem

Every few weeks a new model claims a new high score on a public coding benchmark. The trouble is that public benchmarks are, by definition, public — their problems and expected solutions end up in scraped training corpora, and a model that has effectively seen the answer key posts a number that says more about contamination than capability. For anyone actually choosing a model to build on, the headline benchmark score is close to noise.

What a builder needs is a different question answered: given a task this model has never seen, and a test suite it can't inspect, does the code it writes actually pass? That is a much harder thing to game, and a much more useful thing to know.

The approach

VERITAS scores models on hidden-test pass rate across the full suite. A model is handed a task, produces a solution, and that solution is run against tests it never saw — the pass rate on those hidden tests is the score. There is no partial credit for confident-sounding prose and no reward for having memorized a public answer; the code either passes the tests or it doesn't.

The harness surfaces that in three complementary views:

• Leaderboard — models ranked by hidden-test pass rate across the whole suite, so the headline comparison is one number that's hard to fake.
• Coverage matrix — a per-category grid showing where each model actually holds up and where it falls apart, because an aggregate score hides that a model can be excellent at one class of task and useless at another.
• Task explorer — drill into individual tasks to see how models handled a specific problem, which turns a ranking into something you can actually inspect and trust.

Why it matters

Model selection is now a real engineering decision with real cost attached — the wrong choice means slower delivery, higher token spend, or code that quietly fails in ways a demo never surfaces. A harness that measures capability on unseen work, rather than reputation on seen work, is the difference between choosing a model on evidence and choosing it on a press release. VERITAS is the tool I reach for when that decision has to be defensible.

Status

VERITAS is live and explorable — the leaderboard, coverage matrix, and task explorer are all up and running. The same harness pattern is what I build for teams that need a private, contamination-resistant evaluation of models against their own tasks rather than a generic public suite.