Multimodal Models

An expense app needs one feature. A user takes a photo of a receipt, and the app fills in the store, the date, and the total. The team has not yet chosen between two designs.

• Option A: three tools in a chain. One reads the words in the photo, one cleans up the text, and one sorts it into fields.
• Option B: one model that reads the photo. The app sends the image with the request "Return the store, date, and total as JSON."

In a first test, option B worked on most receipts. On one faded receipt, however, it read a total of 18.70 as 13.70, with no sign of doubt