Grokking LLM Evals
Vote

0% completed

​

Judge Biases

In the MT-Bench study of LLM judges, the researchers showed a judge two answers to the same question and asked which was better. Then they swapped the two answers and asked again. GPT-4 picked the same answer both times in only 65 percent of cases. Claude-v1 did so in 23.8 percent, and it chose whichever answer came first 75 percent of the time.

Nothing about the answers changed between the two questions. Only their order did, and the verdict moved with it.

A judge that responds to something other than the quality of the output has a bias

.....

.....

.....

Like the course? Get enrolled and start learning!

Reading Progress

0%

Vote for new content