Claude Computer Use

2024-10-22 · Publisher : Anthropic · Domain : Artificial Intelligence

Anthropic opens computer use in public beta: Claude looks at the screen, moves the cursor, clicks and types. Announced with an upgraded Claude 3.5 Sonnet (49% on SWE-bench Verified) and Claude 3.5 Haiku.

Anthropic introduces computer use, a capability of the upgraded Claude 3.5 Sonnet that lets it use a computer the way a person does: looking at the screen, moving the cursor, clicking buttons and typing text. The announcement comes with a new version of Claude 3.5 Sonnet and with Claude 3.5 Haiku. ### What changes - Claude receives screenshots and returns actions (mouse movement, click, keyboard input): it can operate any software without a dedicated API. - Public beta in the API: Anthropic describes it as still experimental, at times cumbersome and error-prone. - Limitations acknowledged by the vendor: scrolling, dragging and zooming remain difficult. - The upgraded Claude 3.5 Sonnet makes clear gains in agentic coding and tool use, with no price increase. - Claude 3.5 Haiku, the fast model, matches or beats the original Claude 3.5 Sonnet on several evaluations. ### Specifications - Available in the Anthropic API, Amazon Bedrock and Google Cloud Vertex AI, with a reference implementation published on GitHub. - Tools provided to the model: computer control, text editor and bash terminal. - Claude 3.5 Haiku: announced for later in October, text only at first, with image input to follow. - Precautions: the vendor recommends starting with low-risk tasks and has developed classifiers that detect when the capability is used and possible misuse (spam, misinformation, fraud). ### Pricing | Model (USD per million tokens) | Input | Output | | --- | --- | --- | | Upgraded Claude 3.5 Sonnet (unchanged) | 3 | 15 | | Claude 3.5 Haiku (after the December 3, 2024 revision) | 0.80 | 4 | ### Benchmarks published by the vendor Scores published by the vendor (October 2024 addendum to the model card). | Benchmark | Upgraded 3.5 Sonnet | 3.5 Haiku | 3.5 Sonnet (June) | | --- | --- | --- | --- | | OSWorld (screenshots only) | 14.9% | not disclosed | not disclosed | | SWE-bench Verified | 49.0% | 40.6% | 33.4% | | TAU-bench, retail | 69.2% | 51.0% | 62.6% | | TAU-bench, airline | 46.0% | 22.8% | 36.0% | ### Why it is a turning point Until then, an agent had to go through APIs or integrations written for it. With computer use, the graphical interface designed for humans itself becomes a field of action for AI: it is the starting point for agents that browse and operate applications. ### Key takeaways A score still modest compared with humans, but a shift in approach that other major vendors would adopt in the following months.

Tags : Agents, Anthropic, Claude, Computer Use, Claude 3.5, Claude 3.5 Haiku, Interface graphique