Grokking LLM Evals
Vote

0% completed

​

Final State Checks

The function below grades a run of the scheduling agent without reading a single word the agent wrote.

def check_final_state(task, state):
    bookings = state["bookings"]
    if len(bookings) != 1:
        return False, f"expected 1 booking, found {len(bookings)}"
    tech = TECHNICIANS[bookings[0]["tech"]]
    if tech["trade"] != task["trade"]:
        return False, f"booked a {tech['trade']} technician"
    if not inside_windows(bookings[0], task["windows"]):
        return False, "booking is outside the customer's windows"
    return True, "booking follows the rules"

.....

.....

.....

Like the course? Get enrolled and start learning!

Reading Progress

0%

Vote for new content