Copilot Vision brings Microsoft's screen-watching AI to everyday Windows tasks

[Alfonso Maruccia]

AI overload: Microsoft wants Copilot to be your digital sidekick, always watching and ready to help – whether you need it or not. The latest feature, Vision, turns the AI into a screen-reading assistant that talks back, offering tips, tasks, or tech-induced headaches.

Microsoft announced that Copilot Vision is now available to users in the US, with plans to expand to more non-European countries soon. This update significantly enhances Copilot, which is evolving into more of a digital "companion" for everyday tasks – hallucinations and all. For those wary of intrusive AI "solutions" chasing problems that don't exist, Copilot Vision remains an opt-in feature – for now.

Copilot Vision sees what users see and helps them get things done, Microsoft said on the Copilot blog. Available through the Copilot app, it can navigate multiple application windows at once. Users can ask Copilot to perform specific tasks or offer suggestions, and the AI will either comply or invent a convincing-sounding solution. It's glorious – if you've already accepted living in an AI-slop-filled world.

Copilot Vision also includes a new "Highlights" feature that shows specific steps to complete tasks. It can assist with playing games, viewing photos, and adjusting lighting to enhance colors. Copilot can talk too, conversing in real-time with a synthetic voice, just like the classic dystopian movies that shaped our vision of the future.

Never mind that many daily computer tasks require little actual "intelligence," or that most Windows users are perfectly capable of searching the web on their own. Microsoft insists Copilot makes everything better – and this is how far it's willing to go to embed large language models deeply into Windows.

Microsoft began testing Copilot Vision earlier this year, and the feature is now available on Windows 10 and 11 through the Copilot Labs program – no Copilot Pro subscription required. Users are said to remain in control of the new app-sharing option and can supposedly stop the AI from peeking at their personal data at any time.

Copilot doesn't capture desktop screenshots every few seconds like Recall does, so it poses less of a threat to privacy and data confidentiality – at least for now. Still, any tool that watches the screen and processes sensitive information in real-time is bound to raise security concerns. Given Microsoft's recent track record, users have every reason to remain cautious.

Permalink to story:

Copilot Vision brings Microsoft's screen-watching AI to everyday Windows tasks

 

[Squid Surprise]

Wow, this article isn't biased in any way...

 

[themastergoose]

Did I miss something Techspot?

You weren't ok with static screenshots every 3 seconds but are ok with constant screen monitoring and input tracking, and go as far to say it's less of a privacy concern?

Wow.

 

[Alfonso Maruccia]

Did I miss something Techspot?

You weren't ok with static screenshots every 3 seconds but are ok with constant screen monitoring and input tracking, and go as far to say it's less of a privacy concern?

Wow.

Yes. Being a locally running, pre-trained LLM, this thing shouldn't be a privacy nightmare as much as Recall is. I mean, if you are ready to run an LLM on your computer, and give that thing full web access, who am I to say anything about that....

 

[seeprime]

Did I miss something Techspot?

You weren't ok with static screenshots every 3 seconds but are ok with constant screen monitoring and input tracking, and go as far to say it's less of a privacy concern?

Wow.

I believe the article also said users should remain cautious.

 

[NumberNine]

Yes. Being a locally running, pre-trained LLM, this thing shouldn't be a privacy nightmare as much as Recall is. I mean, if you are ready to run an LLM on your computer, and give that thing full web access, who am I to say anything about that....

I'm more concerned with the local telemetry components talking to the model than the model itself calling home. Middle men. Data collectors. Centralized, even.

 

[OortCloud]

It such a sledgehammer approach to trying to record what you have been doing. If MS built decent journaling support into the OS and got applications to support this and had a search facility that actually worked you wouldn't have to constantly monitor the screen and try to second guess from this what a user is doing. It really is one of the worst (of many really bad) examples of trying to ram these (completely non AI) LLM's into every possible place to justify the massive money you have thrown at their development and on the crazy data centres you are building to train them in.

 

[ScottSoapbox]

Yeah, I keep uninstalling Copilot.

Microsoft likes to reinstall it with updates and then I just remove it again.

I don't have an issue with AI. I have an issue with uninvited installing of intrusive AI.

 

[DrSuess]

I have disable AI functions with a group policy. AI at this time doesn't fit my workflow.

 

[Mr Majestyk]

I uninstalled copilot from my Windows 10 and 11 PC's. Happier days.

 

[brucek]

Does it cost money to buy or use? Because if not, then we know who the product is.

 

[Eldrach]

I really like the idea if having Microsoft track every input I do - including passwords, bank details etc. - absolutely no concern that this info could be intercepted by nefarious beings at all

 

[yannus]

Does it cost money to buy or use? Because if not, then we know who the product is.

Of course they'll spy on users. But what amazes me the most is that nobody cares about how much impact massive adoption will have on power plants! What a paradox, people obsessed with so called ecology but never fighting the real threats.

 

[Theinsanegamer]

Of course they'll spy on users. But what amazes me the most is that nobody cares about how much impact massive adoption will have on power plants! What a paradox, people obsessed with so called ecology but never fighting the real threats.

Because in the grand scheme of things the impact is nothing. It's like looking at the impact a drop of rain will have on a dirt hill. We're pushing EVs hard, the average american drives 40 miles per day. If they only drove 10, they would still use, on average, 2-3 kW per day to drive. That's more energy then enabling copilot on their PC would use in an entire year.

This is the same company that is condemning tens of millions of functional PCs to early graves for entirely arbitrary reasons, replacing those will cause a far greater ecological impact, on both a per user and per capita scale.

I uninstalled copilot from my Windows 10 and 11 PC's. Happier days.

I uninstalled Windows from my PCs. Exuberant Weeks!

Because of this:

Yeah, I keep uninstalling Copilot.

Microsoft likes to reinstall it with updates and then I just remove it again.

I don't have an issue with AI. I have an issue with uninvited installing of intrusive AI.

We have no expectation that MS will respect our choices. They'll reenable copilot, reinstall it, run it in the background, ece.