Evaluating LLM Systems

The Problem Nobody Had Before

Predictive systems can use observed outcomes such as clicks or delivery times, but those labels can still be delayed, incomplete, or biased.

Generative systems add another challenge: several different outputs may all be acceptable.

"Summarize this meeting" has a thousand acceptable outputs and no single correct label anywhere. "Was this support answer good?" is a judgment call, not a fact you can go observe.