Can AI Use Your Computer?

Yes, with your permission, and nowhere near as reliably as the demonstrations suggest. Whether it touches your own machine depends on which of three arrangements you are actually using.

The fear underneath the question is that a chatbot could reach out of the window and start moving your mouse. It cannot. Understanding why takes one paragraph, and everything useful about this subject follows from it.

Lathic has articles on what an AI agent is and what an agentic workflow looks like. This one is about the specific case where the tool being used is a computer.

The model has no hands

Three developers describe the same architecture in their own documentation, independently.

The model is shown a screenshot. It returns structured data naming an action: click at these coordinates, type this text, scroll. It does not perform the action. A separate program, running on a machine, reads that instruction and carries it out.

Google's documentation says the model "analyzes the screen and the prompt, returning a response which includes a suggested function call representing a UI action," and that a developer "will need to implement the client-side execution environment to receive and execute the Computer Use actions." Anthropic's says the application "must receive Claude's tool use requests, translate them into actions in your computing environment, capture results, and return results to Claude." OpenAI's is the shortest of the three: "You provide the environment and execute the model's requests."

So the chain is: screenshot in, proposed action out, separate software executes it, new screenshot, repeat. The loop continues until the model stops asking for actions.

If nobody installed and ran the executing program, nothing happens. There is no path from a chat window to your mouse.

The actions themselves are ordinary input events, not privileged commands. One published toolset lists actions of exactly this kind: screenshot, zoom, drag, mouse move, scroll, type, key, hold key, wait, and the various clicks. These are the same events your keyboard and trackpad generate. In at least one of these systems the coordinates the model returns are in the pixel space of the image it was shown, and the application has to rescale them to the real display; another normalizes them to a fixed grid instead. Either way the point holds: the picture really is the interface. The system is looking at your screen the way you do, not reading the program underneath it.

Three different things called the same name

"AI using a computer" covers at least three architectures with quite different implications.

The third is the one people picture and the least like their situation. Anthropic's documentation states that computer use "requires a sandboxed computing environment," and the reference implementation is a Docker container running a virtual display server, a window manager and preinstalled applications. That is a machine constructed for the purpose. It is not somebody's laptop.

For the browser extension, the requirements are concrete and worth spelling out because they are reassuring without being complacent. You go to the Chrome Web Store, install it, sign in with an account, and grant a named permission list that includes reading and modifying page content, managing tabs, downloads and browser debugging. Then you grant access per site. That is four deliberate acts, and the permission list is visible before you agree to it.

The developers themselves tell integrators to go further. Anthropic's guidance recommends "a dedicated virtual machine or container with minimal privileges," "avoiding giving the model access to sensitive data, such as account login information," and "limiting internet access to an allowlist of domains." OpenAI's is shorter: "use an isolated browser or VM and an allow list of sites and actions."

What it can see

One line from Anthropic's help documentation does more work than any security analysis: "Whatever is visible in one of those tabs is captured in the screenshots and becomes part of the conversation. Claude can't filter sensitive content out of what it sees."

That is the honest shape of the privacy question. The agent does not selectively read. It looks at the screen. Anything on it goes to the model.

Credentials

The documented pattern is that you type the password, not the agent.

OpenAI's documentation states that when a task requires a login, the agent "will pause and prompt you to take control of the virtual browser," and that while you have control, "screenshots are not captured, which helps protect passwords and other sensitive data you enter." There is also a logged-out mode where the agent works without being signed in to sites at all.

Anthropic publishes a list of things its browser agent will not do regardless of permission: making purchases or financial transactions, creating accounts, handling sensitive credit card or ID data, giving financial advice, executing trades, downloading files from untrusted sources, permanent deletions, modifying system files, and completing instructions found in emails or web content. The list restricts actions rather than whole categories of website, which is a meaningful difference: the agent can be on a banking site and still be barred from transacting there.

Google's documentation goes further and instructs the model directly, in capitals, that it is forbidden from accepting or agreeing to terms of service, privacy policies, cookie banners, CAPTCHAs, financial transactions or sending communications on the user's behalf.

Approval gates

The standard architecture puts a human in front of anything consequential, and the categories are enumerated rather than left to judgment.

Google's system returns a safety decision for each proposed action, marking it allowed, requiring confirmation, or blocked, across categories including financial transactions, sensitive data modification, communication tools, account creation, and agreeing to legal terms. Anthropic instructs integrators to ask "a human to confirm decisions that might result in meaningful real-world consequences and any tasks requiring affirmative consent, such as accepting cookies, completing financial transactions, or agreeing to terms of service."

There is a failure mode in the gate itself that is worth naming. OWASP's list of agentic security risks includes what it calls human-agent trust exploitation, where "confident, polished explanations misled human operators into approving harmful actions." An approval gate only works if the person reads it.

The problem nobody has solved

The agent reads web pages. Web pages contain text written by strangers. The model cannot reliably tell instructions from content.

The UK's National Cyber Security Centre explains why this is structural rather than a bug: "Under the hood of an LLM, there's no distinction made between 'data' or 'instructions'; there is only ever 'next token'." Its conclusion is uncomfortable and worth quoting in full: "it's very possible that prompt injection attacks may never be totally mitigated in the way that SQL injection attacks can be." NCSC prefers to describe a language model as an "inherently confusable deputy" rather than a system with a fixable injection bug.

NIST's own guidance takes the same line: because current mitigations "do not offer full protection against all attacker techniques," designers should assume "prompt injection attacks are possible if a model is exposed to untrusted input sources."

Tools raise the stakes rather than changing the mechanism. NCSC again: when a system calls tools based on the model's output, "an attacker can coerce it to use those tools/APIs," raising the impact to "whatever the worse case scenario would be of giving an attacker direct access to those tools/APIs."

The most useful mental model NCSC offers, which it credits to an outside practitioner: when a model processes information from a party, treat its privileges as dropping to that party's level. If it reads an email from a stranger, assume for that moment it is acting for a stranger.

This is not hypothetical. Anthropic disclosed a case from its own testing, before new defenses were applied: "a malicious email claiming that, for security reasons, emails needed to be deleted. When processing the inbox, Claude followed these instructions to delete the user's emails without confirmation."

Why you should distrust every attack-rate number

Anthropic published figures for its browser extension in August 2025: a 23.6% attack success rate without mitigations, reduced to 11.2% with them, and 35.7% reduced to zero on a challenge set of four browser-specific attack types. The company paired this with its own caveat, that internal testing "can't replicate the full complexity of how people browse in the real world."

Then the US government measured the same kind of thing. NIST's Center for AI Standards and Innovation tested an agent in a peer-reviewed injection benchmark and found baseline attacks succeeded 11% of the time. Its own stronger red-team attacks succeeded 81% of the time. Across five injection tasks, success was 57% on a single attempt and 80% on average when the attacker was allowed 25 attempts.

Same agent, same framework. The number moved from 11% to 81% because somebody tried harder.

That is the fact to carry away. Any single attack-success figure describes the test set, not the product. The developers agree: OpenAI says prompt injection "remains a frontier, challenging research problem" and that, "just like traditional scams on the web, we expect our work to be ongoing."

How well does it actually work

Badly, by the standards the demonstrations imply, and this is where the durable point is more useful than any score.

On a peer-reviewed 2024 benchmark of 369 real desktop tasks, a human baseline scored 72.36% and the best model 12.24%. On a 2024 web benchmark of 812 tasks, humans scored 78.24% and the best agent 14.41%. The desktop baseline is worth reading carefully: it came from computer-science undergraduates with basic software skills who had not seen the tasks or the applications before, at a median of under two minutes a task. That is not expert performance, and it is not a general population either.

Those are 2024 figures on 2024 benchmarks. The more recent evidence is more interesting than any average score anyway. A 2026 preprint benchmarking 108 long-horizon workflows, where the median task takes a person about an hour and a half, found the best agent completing 20.6% of tasks outright while scoring 54.8% on partial credit. The gap between those two numbers is the finding: the agents get a long way into the work and then do not finish it. The authors summarized the failure modes: agents "lose track of constraints, miss information that arrives mid-task, guess rather than ask the user, and skip verification, struggling most when a task hinges on hidden state they must recover."

That shape holds even as the numbers move. These systems degrade with task length, and they degrade by quietly proceeding rather than by stopping to ask.

One benchmark caveat is worth surfacing because it applies to all of them. The authors of the desktop benchmark note that their scoring measures whether the task was completed and pays "little attention to potential unnecessary damaging actions of agents." Success rates tell you whether the job got done. They do not tell you what else happened along the way.

What the security agencies say

In April 2026 six national cyber agencies from the US, UK, Canada, Australia and New Zealand jointly published guidance on adopting agentic AI. Its recommendations are incremental deployment, continuous assessment against evolving threats, explicit accountability, rigorous monitoring and human oversight, and it warns that over-privileged agents amplify the impact of a single compromise.

The UK's NCSC gives the clearest single test anywhere in this material: "If you cannot understand, monitor or contain an agent's actions, it is not ready for deployment." And: "never grant an agent unrestricted access to sensitive data or critical systems."

Its controls are the ordinary ones: least privilege, giving an agent "only the minimum access they need, for the shortest time required"; avoiding long-lived credentials; and constraining what it can access, what it can do, and when.

On accountability, NCSC is blunt that it does not transfer. Humans remain answerable for the decision to deploy, the access granted, the safeguards, and the consequences. Anthropic says the same thing to consumers: "You remain responsible for all browser actions taken by Claude performed on your behalf."

The case for it

It would be one-sided to leave it there.

The clearest documented benefit is accessibility, and it has been studied properly. A paper accepted to a 2026 natural language processing conference reports a three-week diary study with eight blind users running 1,258 commands across twelve applications using a screen-reader-accessible computer-use agent. The best model succeeded on 52.5% of commands. The failure modes were the familiar ones: grounding, planning, tracking constraints and knowing when to stop.

About half. That is neither a dismissal nor an endorsement, and the researchers' interviews found that users wanted things beyond automation. But it is a real problem being worked on with the people it would help, which is more than can be said for most of the demonstrations.

What to check before you let one run

• Where does it actually run? Your browser, a hosted browser, or a container. Only the first touches your machine.
• What permissions did you grant? Read them at install time, because that is the moment you have leverage.
• What is on screen? Anything visible goes to the model, and the agent cannot filter it.
• What needs confirmation? Anything hard to undo: sending, paying, publishing, deleting.
• What is it reading? If it reads email or web pages, assume that content can try to instruct it.
• Can you watch it and stop it? If not, by the UK NCSC's own test, it is not ready.