Microsoft wants to bring generative AI to the forefront of Windows — and the computers that run it.
On two keynotes at its annual Build developer conference this week, the company unveiled a new range of Windows machines it calls Copilot+ PCs, plus AI-powered generative features like Recall, which helps users find apps, files and other content they have viewed in the past. Copilot, Microsoft’s brand of generative AI, will soon be much more deeply integrated into the Windows 11 experience. And new Microsoft Surface devices are on the way.
We’ve rounded up all the key messages from Monday and Tuesday here.
Volumetric applications
Microsoft is bringing Windows Volumetric Apps — basically space-oriented, interactive VR apps — to the Meta Quest headset. Through a partnership with Meta, Microsoft says it will deliver Windows 365 and local PC connectivity to the Quest headset, allowing developers to extend their applications into 3D space.
During Tuesday’s keynote, Microsoft showed off a digitally augmented 3D view of an Xbox controller from the perspective of a Meta Quest 3 headset — a digital object that the wearer can manipulate with their hands. “We are deepening our partnership with Meta to make Windows a premium experience on Quest devices,” Pavan Davuluri, CVP of Windows and Devices at Microsoft, said during the demo.
Developers can sign up for a preview to gain access to Microsoft’s new volume API.
Copilot+ computers
Copilot+ PCs are Microsoft’s vision for an AI-first, flagship Windows hardware. All include special chips called NPUs to power AI experiences like Recall. And they come with a minimum of 16GB of RAM paired with SSD storage.
The first Copilot+ PCs will feature Qualcomm’s Snapdragon X Elite and Plus chips, which Microsoft claims deliver up to 15 hours of web browsing and 20 hours of video battery life. Chipmakers Intel and AMD are also committed to creating processors for Copilot+ devices in partnership with a number of manufacturers, including Acer, Asus, Dell, HP, Lenovo and Samsung.
Copilot+ computers start at $999, and some are available for pre-order today.
Surface Pro and Surface Laptop
Microsoft’s recently introduced Surface devices, the Surface Laptop and Surface Pro, focus on performance and battery.
The latest Surface Laptop — available with a 13.8- or 15-inch display — has been redesigned with “modern lines” and thinner screen bezels. It lasts up to 22 hours on a charge and is up to 86% faster than the Surface Laptop 5, the company says. It also supports Wi-Fi 7 and has a touchpad for haptic feedback.
As for the new Surface Pro, Microsoft says it’s up to 90% faster than the previous generation Surface Pro (Surface Pro 9) and features a new OLED with HDR display, Wi-Fi 7 (and optional 5G) and improved ultra-wide front camera. In addition, its removable keyboard – which is reinforced with additional carbon fiber – now has haptic feedback.
Recall
Windows 11’s upcoming recall feature can “remember” apps and content a user accessed on their PC weeks or even months ago, such as helping them find a Discord chat where they discussed clothes they were considering buying. Users can use Recall’s history to “roll back” to see what they’ve worked on in the recent past and break down files like PowerPoint presentations to show information potentially relevant to their searches.
Microsoft says Recall can make associations between colors, images, and more to let users search for virtually anything on their PCs in natural language (not dissimilar to Rewind’s technology at launch); developers will be able to enhance Recall by adding contextual information to their applications. And Microsoft claims that all user data associated with Recall is kept private and on-device — and not used to train AI models, which is important.
Here’s more from Microsoft: “Your snapshots are yours; they remain locally on your computer. You can delete individual snapshots, adjust and delete time ranges in Settings, or pause at any time right from the system tray icon on your taskbar. You can also filter apps and websites from being saved.”
Image editing and live translations
There’s now more AI in Windows than ever before, and some of it exclusive to the new Copilot+ PCs.
A new feature called Super Resolution can restore old photos by auto-scaling. And Copilot can now analyze images to give users ideas for creative compositions. Through a feature called Cocreator, users can generate images and also ask the AI model to follow what they draw to change or restyle the image.
Elsewhere, Live Captions with Live Translations translates any audio that passes through a computer—whether from YouTube or a local file—into the language of the user’s choice. Live translations will initially support around 40 languages, including English, Spanish, Mandarin and Russian.
A separate but related new feature in Microsoft Edge offers real-time video translation of sites like LinkedIn, YouTube, Coursera, Reuters, CNBC, Bloomberg and more. Set to be available in the near future, the feature — which supports Spanish-to-English and English-to-German, Hindi, Italian, Russian, and Spanish — translates spoken content through dubbing and live subtitles.
Team Copilot and extensions
Team Copilot is the latest expansion of Microsoft’s growing Copilot suite of generative AI technologies. It integrates with Teams, the company’s video conferencing app, to help manage meeting agendas and take notes that anyone in a meeting can co-author. And it extends to Loop and Planner, Microsoft’s collaboration and planning platforms, to create and assign tasks, track deadlines, and notify team members when their input is needed.
In somewhat related Copilot news, Microsoft has released (in private preview) Copilot Extensions, which allow developers to extend GitHub’s GitHub Copilot code generation tool with third-party apps and skills. Launch partners include DataStax, Docker, and LambdaTest; extensions will exist in the GitHub Marketplace, but developers will also be able to create their own private extensions to integrate with their internal systems and APIs.
Windows Copilot Runtime
Powering capabilities like Recall and Super Resolution is the Windows Copilot Runtime, a collection of ~40 generative AI models that make up what Microsoft describes as a “new layer” of Windows. In tandem with the semantic index, a vector-based system local to an individual PC Copilot+, the Windows Copilot Runtime allows generative AI applications, including third-party applications, to run without necessarily needing an Internet connection.
“[The runtime] consists of ready-to-use AI APIs such as Studio Effects, Live Captions translations, OCR, Recall with user activity and [more]which will be available to developers in June,” Davuluri said on Tuesday.
Microsoft says CapCut, the popular video editor from TikTok owner ByteDance, will use Windows Copilot Runtime and the accompanying new Windows Copilot Library, a set of APIs and AI development tools, to accelerate its AI features. And Meta will add the aforementioned Studio Effects to WhatsApp to provide features like background blur and eye contact during video calls.
Upgraded bot creators
Azure AI Studio, the set of tools within Microsoft’s Azure OpenAI service that allows customers to combine an AI model and create an application that “reasons” against that data, will soon allow developers to build applications using an inference API for checkout – APIs through which developers can access and fine-tune generative AI models hosted on Azure infrastructure. Microsoft calls this a “model as a service” and is launching with models from Nixtla and Core42 to start with.
In the adjacent Copilot Studio product suite, Microsoft is releasing Copilot agents, which the company describes as AI bots that can “independently organize tasks tailored to specific roles and functions.” (Copilot Studio provides tools to connect Copilot for Microsoft 365, an AI-powered “copilot” in applications like Excel and Word, with third-party data.) Using memory and context knowledge, Copilot agents can navigate different types of business workflows, learn from user feedback, and ask for help when they encounter situations they don’t know how to handle.
Snapdragon Dev Kit
There’s a new development kit from Qualcomm aimed at developers building apps for Copilot+ PCs with an Arm chip
The $899.99 Snapdragon Dev Kit for Windows, which incidentally is the same width, height, and length as Apple’s Mac Mini, packs Qualcomm’s Snapdragon X Elite chip, paired with 32GB of RAM, 512GB of storage, and plenty of I/O. The Dev Kit supports Wi-Fi 7 and Bluetooth 5.4, and through its various USB-C and HDMI ports, it can control up to three 4K monitors at once.
Fi-3
Microsoft has announced an addition to its Phi family of generative AI models, Phi-3-vision, which can perform general visual analysis and reasoning tasks such as answering questions about diagrams and images. The model can read both text and images and is efficient enough to run on a mobile device.
Phi-3-vision is available in preview, while the previously announced text analogs of the model — Phi-3-mini, Phi-3-small and Phi-3-medium — are now generally available.
A partnership with Khan Academy
Microsoft is teaming up with Khan Academy to donate access to cloud computing infrastructure, allowing Khan Academy to offer educators in the US free access to Khan Academy’s AI-based tools. The two companies will also collaborate to explore opportunities to improve AI applications for math lessons through generative AI, Microsoft said on Tuesday.
We’re launching an AI newsletter! register here to start receiving it in your inboxes on June 5th.
