Claude AI Can Now Control Your Computer Screen, Keyboard And Cursor

On Tuesday, Anthropic launched two updated versions of its large language model — Claude AI. Perhaps more interestingly, it also enabled enhanced functionality that allows users to grant the LLM access and control over certain aspects of your personal computer if you choose.

Anthropic has dubbed this new computer navigating capability for Claude — computer use. The company stated in its announcement that this is a new approach toward training AI with help from the public.

“We’re teaching it general computer skills—allowing it to use a wide range of standard tools and software programs designed for people. Developers can use this nascent capability to automate repetitive processes, build and test software and conduct open-ended tasks like research,” the statement reads.

Claude AI’s Computer Use Capabilities Explained

The company’s announcement included a demo video that showed various functionalities of the software responding to a typed task request from a human. In the demo, once Claude was downloaded to a computer, it was able to do the following:

• Take multiple screenshots of a computer screen and analyze the images based on the query request.
• It was then able to search for relevant requested information on the demo computer. When it couldn’t locate the requested details on an open file, it accessed other possible databases or sources as instructed in the human-generated request.
• Once the information was found, the LLM automatically entered the relevant details into a third-party form/application that was open on the screen.
• Lastly, the Claude computer use model went ahead and automatically submitted the completed document.

All steps were accomplished without any intervention by the user once they established the initial request query on their own desktop.

This is important because it’s a significant leap in AI automation access for average individuals, which could further accelerate adoption of AI agents to autonomously complete tasks for us in the future.

Here’s what it looks like in action.

The Anthropic announcement states that this new utility is available for free for interested users who proactively access Claude’s application programming interface for these features and give the LLM access to a personal computer.

Anthropic reiterates several times in its statement that since this software is so new users can expect it to make some mistakes in these early days and be a bit buggy.

“While we expect this capability to improve rapidly in the coming months, Claude’s current ability to use computers is imperfect. Some actions that people perform effortlessly—scrolling, dragging, zooming—currently present challenges for Claude and we encourage developers to begin exploration with low-risk tasks,” the statement reads.

Potential Risks Of Claude AI’s Computer Use

While this AI advancement has created a lot of excitement among AI influencers and developers across social media, there are inherent risks that could arise that Anthropic points out in its post.

“Because computer use may provide a new vector for more familiar threats such as spam, misinformation, or fraud, we’re taking a proactive approach to promote its safe deployment. We’ve developed new classifiers that can identify when computer use is being used and whether harm is occurring. You can read more about the research process behind this new skill, along with further discussion of safety measures, in our post on developing computer use,” the company wrote.

Anthropic did not reply to an email inquiry for further comment regarding the Claude AI upgrades prior to publication of this article.