Design a Browser Automation Agent

Imagine helping a relative pay a bill while you are on the telephone and they are at the computer. You cannot see the screen, so you ask them to describe it. You say, "Click the blue button at the top", and they ask, "Which one? There are two now." A window has opened over the page. After they click, you do not know whether the payment went through until they read the next page aloud.

A browser agent works under the same conditions. The model sees only a description of the screen, and it acts through an interface designed for people, not programs