Computer-using AI

Computer-using AI takes a screenshot of a whole machine, decides where to click and what to type, and does it. No API, no integration, no per-application connector — the same interface a person uses.

The appeal is obvious. Every piece of software becomes automatable, including the legacy desktop application with no API that half an industry depends on and nobody will ever modernise.

That generality is genuinely the only thing it offers over every other approach, and it is worth being clear about that. It is not faster, cheaper or more reliable than an integration — it is available where an integration is not.

A browser gives an agent a structured representation — the DOM and accessibility tree — so it can reason about elements rather than pixels. A desktop often gives nothing comparable, so the model works from the image.

That makes everything harder. Coordinates must be precise, windows overlap, the same application looks different on another machine, and the model has no reliable way to tell a disabled button from an enabled one. Error rates are substantially higher than in browser automation for the same reason.

Screen resolution and scaling are an underrated source of trouble. An agent that works reliably on one display can mis-click consistently on another, because a coordinate it learned is no longer where the control sits.

The specific thing that makes this risky is that a mis-click raises no error. A missing DOM element throws; clicking forty pixels too low simply activates something else, and the agent observes a screen it did not expect and tries to make sense of it.

That produces a characteristic failure where the agent recovers into the wrong task. It clicked the adjacent menu item, found itself somewhere unintended, and proceeded — competently — to do something nobody asked for.

The mitigations are behavioural rather than clever. Re-read the screen after every action, prefer keyboard navigation over pointer targeting wherever the interface allows, and treat any unexpected screen state as a stop rather than something to improvise around.

A browser agent is confined to a browser. A computer-use agent has whatever the logged-in user has: files, credentials stores, email client, terminal, everything.

Combine that with prompt injection and the shape of the risk becomes clear. Text on screen is input, and a document or web page the agent reads can contain instructions. An agent that can be influenced by what it reads, and can also open a terminal, is a category of exposure that no prompt instruction contains.

There is no equivalent of tool scoping here, which is what makes it distinctive. With a tool-using agent you can simply not provide a delete tool; with mouse and keyboard control, every capability the machine has is already available and the only boundary left is the machine itself.

• Runs with the full permissions of the logged-in user
• Any text on screen — document, page, email — is potential instruction
• No natural boundary between applications
• Credentials visible on screen are visible to the agent

Every responsible deployment runs the agent in a dedicated virtual machine or container with nothing sensitive on it, given only the specific credentials the task requires, with network egress restricted and the session recorded.

That containment is the product, not an afterthought. Running a computer-use agent on your own working machine, signed into your own accounts, is the configuration to avoid — however convenient the demo made it look.

Egress restriction deserves particular emphasis, because exfiltration is the goal of most injection attempts. An agent that can only reach the two hosts the task requires turns a successful injection into a failed one, regardless of how thoroughly the model was persuaded.

It works, slowly and imperfectly, and improves noticeably with each model generation. For high-value, low-frequency tasks — migrating data out of a system with no export, operating a vendor tool nobody has an integration for — it is already useful.

For high-frequency tasks it remains too slow and too unreliable. The practical rule: if you would build an integration given a week, build the integration. Computer use is for the cases where no integration will ever exist.

The honest framing is a bridge rather than a destination. It is how you automate the thing that will be replaced in three years but has to keep running until then, and treating it as a permanent architecture is how teams end up maintaining something slow, expensive and hard to debug.