Grokking LLM Evals
Vote

0% completed

​

Code Checks for Format and Rules

import json TRADES = {"plumbing", "electrical", "heating_and_cooling", "appliance", "carpentry", "other"} def check_trade(raw_output): try: ticket = json.loads(raw_output) except json.JSONDecodeError: return "fail", "not valid JSON" if ticket.get("trade") not in TRADES: return "fail", f"trade {ticket.get('trade')!r} is not an allowed value" return "pass", ""

These few lines grade one output of the job ticket assistant for two problems: text that is not valid JSON, and a trade outside the six the business uses

.....

.....

.....

Like the course? Get enrolled and start learning!

Reading Progress

0%


Vote for new content