Comparing Against a Reference Answer

A customer writes: "The furnace is not producing any heat and it is below freezing outside. We have two small children." A dispatcher writes the reference summary for this test case: "No heat from the furnace during freezing weather. Two small children in the home."

The assistant produces two summaries on two runs. Summary A reads "Furnace not heating while it is below freezing outside; customer has two young kids." It is correct, yet it shares only three words with the reference. Summary B reads "No heat from the Heatmax furnace during freezing weather. Two small children in the home