Harmful Requests and Jailbreaks

HarmBench, a public test set for attacks on language models, contains 510 harmful behaviors. Its authors used it to compare 18 attack methods against 33 target models and defenses. They also found that one setting of the test itself changes the attack success rate a great deal. That setting is the number of tokens the target model may generate before its reply is graded, and papers that did not fix it report rates that can mislead.

That last finding applies directly to a product team. The same replies, graded in two ways, give two different safety numbers