A/B Tests for LLM Features

"SRMs typically invalidate the A/B test and make any results and metric movements untrustworthy."

That sentence comes from Microsoft's Experimentation Platform team. An SRM, or sample ratio mismatch, means the two groups in a test are the wrong sizes. They did not receive the share of traffic the test was set up to give them. The team reports that about 6 percent of its own A/B tests have one.

LLM features have a common way of producing this mismatch, and they differ from ordinary product changes in three other ways as well