Retiring Harness Parts When Models Improve

"every component in a harness encodes an assumption about what the model can't do on its own"

That sentence comes from Anthropic's engineering article "Harness design for long-running application development". The article adds that these assumptions "are worth stress testing", because they "can quickly go stale as models improve."

A harness part added to correct a model's weakness keeps costing tokens, time, and maintenance after the weakness has gone, and no error or log line shows that it has gone