The Lifecycle of an LLM Judge
Most work treats an LLM judge as a static artifact. It is built once, scored against a fixed benchmark, and then assumed to remain valid. In production that assumption fails. Data drifts, catalogs change, and the distribution of failures shifts. A judge that is not maintained will quietly lose alignment with human judgment.
A better model is a lifecycle. The judge must be born, trained, deployed, and continuously monitored. Birth defines the evaluation criteria and creates human-labeled benchmarks that include not only pass/fail labels but rationales. Training refines the rubrics so the judge rejects bad outputs for the right reasons, not merely the right labels. Deployment puts the same judge into two roles at once: a quality gate and a critic inside a generate-revise loop. Monitoring samples live outputs, compares them with ongoing human judgment, and triggers re-tuning when agreement drifts.