The Job Ticket Assistant

The job ticket assistant makes five promises:

• Every request that describes a danger to people or property is marked as an emergency.
• Every ticket names the right trade.
• The summary for the technician contains no detail the customer did not give.
• The reply to the customer never promises an arrival time or a price.
• When the address or the problem is missing, the reply asks for it.

It also makes three non-promises. It does not diagnose the fault, it does not quote a price, and it does not book the visit.

Those eight lines are more useful to an eval than any description of the model behind the feature, because each line can become a test. This lesson describes the feature that most examples in this course test, and shows how its promises turn into the parts of an eval.

What the Assistant Does

The job ticket assistant turns a customer's description of a problem into a job ticket, a summary for the technician, and a reply to the customer. It runs inside a home repair booking app that receives about 18,000 requests a day. Before the assistant existed, a dispatcher read each request and filled in the ticket by hand.

The input is the customer's own text, plus two facts the app already knows: the address on the account and the type of home, such as an apartment or a house. The text can be one line or several paragraphs, and customers write in their own words, not in the trade's vocabulary.

What Comes Out

The assistant returns one structured answer. Here is the answer for a request that reads "Our fridge stopped cooling this morning and it is full of food. The ice maker is also leaking."

{
  "trade": "appliance",
  "urgency": "same_day",
  "fields": {
    "appliance": "refrigerator",
    "location_in_home": "kitchen",
    "access_notes": null
  },
  "summary": "Refrigerator stopped cooling this morning and is full of food. Ice maker is leaking.",
  "reply": "Thanks for the details. We have logged a same day repair for your refrigerator. A dispatcher will confirm the visit time shortly."
}

The answer has three parts, and each one serves a different reader:

The six trades are plumbing, electrical, heating and cooling, appliance, carpentry, and other.

Notice that the three outputs go to three different readers, and a mistake in each one costs something different. That difference is why one overall score cannot describe whether the feature works.

What Doing Its Job Means

A promise is useful to an eval only when someone can decide, for any single output, whether it was kept. "The assistant handles emergencies well" cannot be checked. "Every request that describes a danger to people or property is marked as an emergency" can be checked, once the team lists what counts as a danger. The list is short: a smell of gas, sparks or burning, water coming through a ceiling or flooding a floor, and no heat during freezing weather.

Each of the five promises was written this way. Two of them also need a short definition before anyone can grade them:

• The right trade. A leaking ice maker could be plumbing or appliance. The team decides that anything inside an appliance is appliance work, and writes that rule down.
• No invented detail. A summary may shorten and reorder what the customer wrote. It may not add a brand, a model number, an age, or a cause the customer did not mention.

These definitions will change as the team reads more outputs and meets cases nobody expected. The promises are where the eval starts, not where it ends.

What It Does Not Promise

Non-promises matter as much as promises, because they keep the eval from grading the feature on work it was never asked to do. The assistant does not diagnose: "Ice maker is leaking" is a correct summary, while "Water inlet valve has failed" is a guess. It does not quote prices, because prices depend on the technician's visit. It does not book the visit, because a separate scheduling feature does that.

A non-promise also tells the grader what to treat as a failure. A summary that names a likely cause breaks a non-promise, so the grader for invented details marks it as a fail, even when the guess happens to be correct.

From Promises to Evals

Each promise points to two parts of an eval: the test cases it needs and the kind of grader that can check it.