# Claude Computer use Anthropic system to let Claude control your machine and perform various actions (e.g., interacting with your desktop, browser, file system, etc). Claude receives screenshots, reasons about them, and returns clicks, keystrokes, and scrolls to execute. In [[Claude Code]] this surfaces as [[Claude Code Computer Use]]; in the browser as [[Claude in Chrome]]. Anthropic published a detailed best-practices guide for production integrations (see References). The essentials below. ## Resolution and scaling Click accuracy is the foundation of any computer use integration, and unscaled screenshots are the number one cause of missed clicks. Pre-downscale before sending; don't let the API do it for you. - API limits: Claude 4.6 models cap at 1568 px on the long edge and 1.15 megapixels; Opus 4.7 raises that to 2576 px and 3.75 megapixels - Default to 1280x720; on Opus 4.7, 1920x1080 is a real quality bump at acceptable token cost - Avoid native unscaled resolution, anything below 960x540, and macOS screenshots captured at 2x device pixel ratio without adjustment - Scale returned coordinates back to native resolution: `screen_x = api_x * (screen_w / display_w)`, same for y - Put the text instruction BEFORE the image in the content array; the model clicks better when it knows what it's looking for ## Click accuracy troubleshooting - Consistent directional offset: your `display_width_px`/`display_height_px` don't match the actual resized screenshot - Near-misses on dense UIs: enable zoom, reduce source DPI, or crop before downscaling - Wrong element entirely: add positional context to the prompt ("blue Submit button, bottom-right") and break the interaction into smaller steps - What does NOT help: tiling screenshots, overlaying coordinate grids, fancy resize algorithms (LANCZOS vs sips: identical results) ## Model selection - Sonnet 4.6: best mechanical precision and accuracy-to-cost balance; the default - Opus 4.7: clicking roughly on par with Sonnet 4.6 plus stronger reasoning and a higher resolution budget - Haiku 4.5: when latency wins - Orchestrator pattern: a reasoning model plans, Sonnet/Haiku execute the mechanical steps ## Small targets Enable `enable_zoom: True` so the model can inspect a region at higher resolution before clicking. Beyond that: slightly bigger UI elements pay off disproportionately, keyboard shortcuts beat hunting for tray icons, and cropping to the relevant region beats sending the full display. ## Thinking effort UI automation is perceptual and mechanical; more reasoning only helps for planning multi-step sequences and recovering from unexpected states. - Claude 4.6 family: medium effort is the sweet spot (near-max success at about half the output tokens of high). Low effort can beat disabled thinking on total tokens because it makes fewer errors. - Opus 4.7: high effort by default; low effort uses about a tenth of the tokens for high-throughput work; reserve max for one-shot-must-succeed tasks ## Prompt injection defense Layered: training-time robustness (RL against injected web content), real-time classifiers scanning what enters the context window, and continuous red teaming. The official `computer_20251124` tool type activates the classifiers automatically at no extra latency or cost; custom tool implementations do NOT get this. On top of that: human-in-the-loop for irreversible actions, restrictively scoped permissions, logging with screenshots at each step, and treating all web content as untrusted. ## Context management for long runs - Cache breakpoints: one on the stable prefix (system prompt, tools), three rolling on recent tool results - Rolling buffer: keep the N most recent screenshots (default 3), replace older ones with text placeholders in batches so the message prefix stays byte-identical between prune events; this is what keeps [[Claude Code Prompt Caching]]-style cache hits alive - Compaction: summarize before discarding old images (verbatim user instructions, actions taken, errors and fixes, current state, next step); server-side compaction exists in beta via the `context_management` parameter - Anthropic's claim worth remembering: context engineering has more impact on cost and latency than almost any other optimization ## Experimental patterns - Batch tools: one tool call carrying a list of sub-actions (click, type, key presses) to cut round trips; only for self-contained sequences, since one missed click poisons the rest - Advisor tool (beta): a Sonnet executor consults an Opus advisor mid-generation for planning moments and unexpected modals; capped via `max_uses` - Periodic nudges: on long sessions, remind the model every ~20 turns that batch and advisor tools exist ## Teach mode (demonstration-based learning) Show, don't tell: record a human doing the task (actions, annotated screenshots, optional voice narration), then replay the recording as context so Claude follows the same steps while adapting to UI changes. Store both CSS selectors and coordinates; selectors survive layout changes, coordinates are the fallback. Three strictness modes: strict (compliance workflows), adaptive (default), goal-oriented (frequently-changing UIs). ## References - Documentation: https://docs.anthropic.com/en/docs/build-with-claude/computer-use - Best practices for computer and browser use: https://claude.com/blog/best-practices-for-computer-and-browser-use-with-claude - Demo: https://github.com/anthropics/anthropic-quickstarts/tree/main/computer-use-demo#quickstart-running-the-docker-container - Introduction: https://www.youtube.com/watch?v=7TtuiNnhwmM ## Related - [[Claude Code]] - [[Claude Code Computer Use]] - [[Claude in Chrome]] - [[Claude Code Prompt Caching]] - [[AI Agents]] - [[How to use Claude Computer use in a Docker container]]