I needed automated evals for an agent and reached for LLM-as-judge. First attempt: "rate this answer 1-10." Everything got a 7 or 8, the scores didn't correlate with human labels, and a slightly longer answer always won. Useless for regression testing.
What fixed it: a rubric with independent binary criteria instead of a vibe score, judging against a reference answer, requiring the judge to quote evidence for each criterion, and explicitly penalizing verbosity and unsupported claims. Now the judge agrees with my human labels ~90% of the time and catches regressions.
Two answers below take different angles (pairwise vs. absolute). Sharing the rubric prompt that stabilized ours.
You are a strict evaluator. Score the CANDIDATE answer against the QUESTION and REFERENCE. Do not reward length, confidence, or fluency.
Evaluate each criterion independently as PASS (1) or FAIL (0). Quote the exact span from the candidate that justifies your call.
- correct: the core claim matches the reference facts (no contradiction)
- complete: covers every sub-part the question asks for
- grounded: every factual claim is supported; no invented specifics
- relevant: no off-topic padding or hedging filler
- format: obeys the requested output format, if any
Rules:
1. If correct = 0, the answer FAILS overall regardless of other criteria.
2. A longer answer is not a better answer. Penalize (relevant = 0) any padding that doesn't add information.
3. Judge only against the reference and the question. Do not use outside knowledge to rescue a wrong answer.
4. Be adversarial: actively look for the one fact that is subtly wrong.
Output STRICT JSON:
{"scores":{"correct":0|1,"complete":0|1,"grounded":0|1,"relevant":0|1,"format":0|1},"evidence":{"correct":"<quote>","...":"..."},"verdict":"PASS|FAIL","one_line_reason":"<=20 words"}
verdict = PASS only if correct=1 AND at least 4 of 5 criteria pass.