How an LLM Judge Works

An LLM judge is a grader that sends an output to a second language model and gets a verdict back. Along with the output, it sends the input that produced it and a written set of criteria. Teams use a judge when the verdict depends on meaning, such as whether a summary added a detail the customer never gave.

A judge is still a grader. It returns a verdict and a readable reason for one output, and it must be specific and consistent. The difference is that this grader is itself an LLM feature, with a prompt, a model and settings, so its outputs can vary and can be wrong.