Skip to content

Projects / Computer Use

Computer Use Studio

Safer public showcase: realistic replay + full trace + engineering deep dive. It demonstrates an AI agent controlling a desktop (mouse, keyboard, and sandboxed code execution) without exposing a public arbitrary-execution endpoint.

Recorded Session · 1366×768safe replay
Firefox · about:blank
about:blank

Session ready

Desktop initialized, waiting for first action

Frame 1 / 5 · 00:00

Mouse + Keyboard

Tool calls drive deterministic movement, clicks, key combos, and text input.

Sandboxed Bash

Code execution happens in isolated container runtime, not on the host machine.

BYOK Mode

Live private demos support provider key entry in UI, with no public arbitrary access.

Behind the Build

Designing for Trust, Not Just Demos

The strongest lesson from this project: impressive automation means nothing if users can’t trust safety, observability, and cost boundaries.

The technical core is a tool-augmented agent loop. The model receives screenshots and user intent, then emits structured tool calls for mouse movement, clicking, typing, and optional shell execution. The runtime executes those actions in a containerized desktop and returns results back into the model loop. This is what makes the system feel interactive rather than purely conversational.

Reliability work dominated development. I had to harden tool schema semantics, improve argument handling, and tune transcript continuity so malformed or interrupted tool exchanges didn’t collapse sessions. In practice, “agent UX” is mostly protocol quality: if tool calls are brittle, the experience feels random.

Cost control required explicit strategy. Screenshot-heavy loops can explode token usage, so I shifted behavior toward checkpoint verification instead of capturing every micro-step. That single change improved both spend predictability and trace readability.

For portfolio publication, I chose a safer public architecture: replayed runs plus transparent traces. This preserves evidence of real capability while avoiding public arbitrary execution risk. In private demo contexts, the same system can run in BYOK mode with strict controls.