GUI agent
Also known as: GUI-using agent
A GUI agent interacts with graphical interfaces: it perceives the UI (via pixels or the accessibility tree) and issues clicks, typing, and drags.
- Can target any app with a GUI, not just browsers.
- Two modes: vision (screenshots) or DOM/AX-tree (structured).
- The bridge between “no API” and “full computer control.”
The spectrum: tool call < browser agent < GUI agent < full computer use. Each step down in specificity gains reach and loses reliability. Pick the narrowest that gets the job done.