Computer Use is an AI capability that allows language models to see a computer screen, interpret what is displayed, and take actions – clicking buttons, typing text, scrolling pages, opening applications, and navigating menus – just as a human would. Instead of generating text responses, a Computer Use-enabled AI operates the computer directly.

TL;DR – Key takeaways

Definition: Computer Use = screen understanding + cursor/keyboard control + multi-step planning. Not just “the AI sees your screen” – the AI acts on it.
Who has it: Anthropic Claude (via the “Cowork” feature in Claude Desktop) is the only mainstream desktop AI with real Computer Use today. OpenAI Operator is a browser-only preview. Google Gemini has screen share (observation only) but not screen control.
Why it matters: Bridges the “API gap” – most enterprise software exposes no API, so the only way to automate it is to click through the UI like a human.
What it cannot do yet: Reliably handle CAPTCHA, complex drag-and-drop, deeply nested accessibility trees, or long multi-hour workflows without human checkpoints.
Where the risk is: Screen access + input control = a model that can read your passwords and move money. Confirmation prompts, session scoping, and audit logs are essential.
Windows gap: No native Gemini Windows app exists at all, so Computer Use on Windows currently means Claude Desktop or Operator, not Gemini. See our Windows coverage.

What Computer Use does

Traditional AI assistants work through text: you describe what you need, the AI describes what to do. Computer Use removes the description layer. The AI sees your actual screen, understands the interface elements (buttons, text fields, menus, icons), and performs actions on them.

This enables tasks that are difficult or impossible through text alone. Filing an expense report across three different web applications. Copying data from a spreadsheet into a form. Navigating a settings menu that has no API or keyboard shortcut. Performing repetitive GUI operations that would take a human minutes of clicking but cannot be scripted because the interface is not programmatically accessible.

The capability requires two permissions: screen access (the AI must see what is on the display) and input control (the AI must be able to send mouse clicks and keyboard events). These permissions make Computer Use more sensitive than standard AI chat – the model is acting on your system, not just talking about it.

The practical shift is that Computer Use inverts the traditional automation model. Historically, automation required a developer to model every element of the target UI – XPath selectors, accessibility IDs, keyboard shortcuts. Computer Use asks the model to look at a screenshot and decide, in natural language, what to do next. That flexibility is the whole point: if the UI changes, the model adapts; if a new app launches, the model handles it without you updating a script.

How Computer Use works

Computer Use models process screenshots as visual input. At each step, the model receives a screenshot of the current screen state, analyzes the visual content (text, buttons, layout, icons), decides what action to take next, and sends that action (click at coordinates X,Y; type this text; press this key combination). The system then captures a new screenshot reflecting the result and repeats.

This screenshot-action loop is fundamentally different from API-based automation. Traditional automation tools (Selenium, AppleScript, Shortcuts) interact with applications through programmatic interfaces – they know the structure of the UI and can address elements directly. Computer Use interacts visually, the same way a human does. This makes it more flexible (it works with any application that has a visual interface) but less precise (it can misread a button or click the wrong coordinates).

The models behind Computer Use are multimodal – they process both text and images. The vision component identifies UI elements and their states (is this checkbox checked? is this dropdown expanded?). The language component reasons about what action to take given the current state and the user’s goal.

Under the hood, a Computer Use agent loop looks roughly like this:

Perceive: Capture a screenshot. Optionally pull the accessibility tree (on macOS, through AXUIElement; on Windows, through UIAutomation).
Plan: Given the user goal (“file the April expense report in Concur”), decide the next single action.
Act: Emit a structured action – click(x=420, y=118), type("April 2026 conference travel"), scroll(dy=-300), or key("cmd+s").
Observe: Wait for the system to update, then capture a fresh screenshot.
Loop or stop: If the task is complete, return a summary. If not, go to step 2. If a destructive action is pending (send email, delete file, confirm payment), pause for explicit human confirmation.

Modern implementations mix pure visual understanding with structured hints. Claude Sonnet 4’s Computer Use mode, for example, reads the raw accessibility tree when available and falls back to pixel-level screenshot analysis when the tree is absent. That hybrid approach improves reliability on native apps while preserving flexibility on web apps that hide their internals.

Current implementations

Anthropic Claude Computer Use (Cowork)

Anthropic launched Computer Use as a beta feature for Claude in late 2024, making it the first major AI company to offer desktop control through a language model. Claude Computer Use works through an API that developers integrate into their own tools. The model receives screenshots, returns action commands, and the host application executes them.

In 2025, Anthropic productized the feature inside Claude Desktop under the brand name “Cowork”. Claude Desktop’s Cowork mode sits as a separate agent that can drive your cursor, open apps, and complete multi-step tasks – it is the only mainstream consumer desktop AI shipping with this capability today.

Claude’s implementation emphasizes safety: actions are logged, the model asks for confirmation on destructive operations, and the system is designed for developer-controlled environments rather than unsupervised autonomous operation. If you want a head-to-head breakdown of Claude versus Gemini on this and other dimensions, see Gemini for Mac vs Claude Desktop.

OpenAI Operator

OpenAI’s Computer Use product is called Operator. At launch, Operator was browser-only – it runs in a cloud-hosted Chromium instance rather than touching the user’s local desktop. This reduces the security surface at the cost of losing access to native macOS and Windows apps. Operator sits behind the ChatGPT Pro tier and remains a preview in most regions.

Google Project Mariner (Gemini 3)

Google’s Computer Use implementation is called Project Mariner, built on the Gemini 3 model family. Project Mariner focuses on browser-based tasks – navigating websites, filling forms, extracting information from web pages. It represents Google’s approach to agentic AI that operates through existing interfaces rather than requiring dedicated APIs.

Notably, Project Mariner is still an experimental research preview. It is not shipped inside the Gemini consumer app, and it is explicitly not part of the Gemini for Mac release on 2026-04-15. That means no version of Gemini – web, iOS, Android, or Mac – currently includes general-purpose Computer Use as a user-facing feature.

What is not yet available

As of the Gemini for Mac launch, Computer Use is not included in the desktop application. The Gemini Mac app supports text chat, image generation, Canvas, Deep Research, and screen sharing (viewing your screen as context for conversation) – but not autonomous screen control. Screen sharing lets Gemini see and comment on your screen. Computer Use would let Gemini act on your screen. The distinction is observation versus operation.

This absence is notable because the desktop is where Computer Use has the most potential value – automating cross-application workflows, navigating complex enterprise software, performing repetitive GUI tasks. It is also a genuine gap in Google’s otherwise aggressive feature matrix: see our review of Gemini for Mac for the full list of what did and did not ship.

Comparison matrix

Capability	Claude (Cowork)	OpenAI Operator	Google Gemini	Gemini for Mac
Read screen	Yes	Yes (cloud browser only)	Yes (Screen Share)	Yes (Screen Share)
Move cursor	Yes (local)	Yes (cloud browser)	No	No
Click buttons	Yes (local)	Yes (cloud browser)	No	No
Type text	Yes (local)	Yes (cloud browser)	No	No
Run on native apps	Yes (macOS, Win)	No	No	No
Cross-app workflows	Yes	No (browser only)	No	No
Human confirmation on destructive actions	Yes	Yes	N/A	N/A
Availability tier	Claude Pro / Max	ChatGPT Pro preview	N/A	N/A

Real-world use cases

1. Expense reports and procurement

Anyone who has filed a Concur, Expensify, or SAP Ariba report knows these systems expose minimal API surface. Computer Use flies through the forms – reading receipt OCR from your downloads folder, entering vendor names, choosing project codes, and submitting. The human checkpoints are usually on the “submit” and “pay” buttons.

2. Cross-app data transfer

Copying rows from a spreadsheet into a web CRM, pulling names from a LinkedIn search into a Notion database, or translating a PDF and pasting the output into Slack. All of these are classic Computer Use tasks because no single API covers the whole flow.

3. QA and UI testing

Developers use Computer Use agents to exercise UI flows without writing Selenium scripts. Give the agent a user story (“sign up, add three items to cart, checkout with a saved card”) and it executes the flow across a real browser, reporting failures in natural language.

4. Accessibility and kiosk automation

Internal tools, kiosk software, and legacy Windows applications often have no automation hooks. Computer Use is the only practical option for automating these systems.

5. Personal productivity

Booking travel, filling out healthcare forms, onboarding a new SaaS vendor – tasks that look like real work but are mostly clicking. Computer Use is slowly moving this class of work from “things humans do” to “things agents do, with humans confirming.”

Windows context: Computer Use without a native Gemini Windows app

Because Google has not shipped a native Gemini Windows app, Windows users interested in Computer Use have two real options today:

Claude Desktop on Windows: Ships with Cowork. Fully native, available on the Microsoft Store and as a direct download.
OpenAI Operator: Works in the browser regardless of OS. Cloud-hosted, so your local Windows apps stay out of reach.

Gemini’s Project Mariner preview can be accessed through Chrome on Windows, but it is an experimental research surface, not a production product. For a deeper look at what Windows users are missing, see Google didn’t make a Gemini Windows app and our Gemini Windows install guide for the closest alternatives.

Safety and privacy considerations

Computer Use raises specific concerns that text-based AI does not. The model sees everything on your screen, including sensitive content (passwords, personal messages, financial data) that may be visible during a task. Input control means the model can take actions with real consequences – sending emails, deleting files, submitting forms.

Current implementations address this through explicit permission grants (you opt in to screen sharing and input control), action confirmation (the model pauses before destructive actions), session boundaries (Computer Use runs in defined sessions, not continuously), and audit logging (all actions are recorded for review).

A few practical defenses power users apply when giving an AI real desktop control:

Use a dedicated browser profile for agent sessions so cookies and saved passwords are scoped.
Turn on confirmation prompts for destructive actions – most Computer Use implementations default to manual confirmation on “send”, “delete”, and “pay”.
Restrict at the OS level: on macOS, manage Screen Recording and Accessibility permissions per app; on Windows, run the agent in a non-admin account.
Log everything: always enable the agent’s action log so you can audit what the model did after the fact.
Time-box sessions: an agent left running for hours accumulates drift; most tasks should complete within 5-15 minutes of wall time.

Known limitations and edge cases

CAPTCHA and anti-bot walls: Computer Use agents fail on most modern CAPTCHAs by design, and many websites explicitly flag agent traffic.
Latency: A screenshot-action loop runs at 1-4 actions per second. Agents are slower than a fast human operator.
Hallucinated clicks: The model occasionally clicks the wrong button if two UI elements look similar. Good implementations include OCR-level verification.
State drift: Long tasks that involve waiting (e.g., waiting for a page to load) can confuse the agent, which might retry or give up prematurely.
Legal and ToS risk: Automating third-party websites may violate their Terms of Service. Enterprise users should check before building production workflows.

Screen sharing: Gemini’s current capability to view your screen as context – observation without action.
Project Mariner: Google’s Computer Use implementation built on Gemini 3 (research preview only).
Agentic AI: AI systems that act autonomously, of which Computer Use is a specific application.
MCP (Model Context Protocol): Anthropic’s protocol for connecting AI models to tools and data sources – an alternative to visual Computer Use when the target system exposes an API.

FAQ

Does Gemini have Computer Use? Not yet in any consumer product. Project Mariner is Google’s research preview; none of the shipping Gemini apps (web, iOS, Android, or the new Gemini for Mac) include Computer Use.

Can I try Computer Use on my own computer today? Yes, with Claude Desktop on macOS or Windows through the Cowork feature. OpenAI Operator offers a cloud-browser variant. Both require paid tiers.

How is Computer Use different from Screen Share? Screen Share is observation – the model sees your screen and can talk about it. Computer Use is operation – the model sees and also acts on your screen.

Is Computer Use safe for sensitive work? It depends. Regulated industries should treat Computer Use like a junior intern with admin access: useful, but only under supervision. Use dedicated accounts, enable confirmation prompts, and review logs.

Will Google add Computer Use to Gemini for Mac? Google has not announced a timeline. Given Project Mariner is still a research preview, a consumer-grade Computer Use feature is likely 6-12 months out at best.

Can I build my own Computer Use agent? Yes. Anthropic exposes the Computer Use tool as part of the Claude API, and open-source projects like Open Interpreter and Pyautogui-based agents exist. You supply the model and build the execution harness.

Desktop AI without Computer Use

GeminiDesktop provides the Gemini experience on Mac with screen sharing, Canvas, and Deep Research. While Computer Use is not yet available on any Gemini desktop client, GeminiDesktop will support it when Google enables the capability. Download at geminidesktop.app.