Grokking LLM Evals
0% completed
Code Checks for Format and Rules
import json TRADES = {"plumbing", "electrical", "heating_and_cooling", "appliance", "carpentry", "other"} def check_trade(raw_output): try: ticket = json.loads(raw_output) except json.JSONDecodeError: return "fail", "not valid JSON" if ticket.get("trade") not in TRADES: return "fail", f"trade {ticket.get('trade')!r} is not an allowed value" return "pass", ""
These few lines grade one output of the job ticket assistant for two problems: text that is not valid JSON, and a trade outside the six the business uses
.....
.....
.....
Like the course? Get enrolled and start learning!
Reading Progress
0%