Behind the Build
Designing for Trust, Not Just Demos
The strongest lesson from this project: impressive automation means nothing if users can’t trust safety, observability, and cost boundaries.
The technical core is a tool-augmented agent loop. The model receives screenshots and user intent, then emits structured tool calls for mouse movement, clicking, typing, and optional shell execution. The runtime executes those actions in a containerized desktop and returns results back into the model loop. This is what makes the system feel interactive rather than purely conversational.
Reliability work dominated development. I had to harden tool schema semantics, improve argument handling, and tune transcript continuity so malformed or interrupted tool exchanges didn’t collapse sessions. In practice, “agent UX” is mostly protocol quality: if tool calls are brittle, the experience feels random.
Cost control required explicit strategy. Screenshot-heavy loops can explode token usage, so I shifted behavior toward checkpoint verification instead of capturing every micro-step. That single change improved both spend predictability and trace readability.
For portfolio publication, I chose a safer public architecture: replayed runs plus transparent traces. This preserves evidence of real capability while avoiding public arbitrary execution risk. In private demo contexts, the same system can run in BYOK mode with strict controls.