Blog

If you’ve been considering buying a new laptop, you’ll no doubt have noticed that they’re increasingly boasting NPU capabilities that sound an awful lot like the hardware we’ve seen in top smartphones for a few years. The driving factor is laptops’ drive to catch up with mobile AI capabilities by embedding them with advanced AI features, such as Microsoft’s Copilot, that can run securely on the device without needing an internet connection. So here’s everything you need to know about NPUs, why your next laptop might have one, and whether or not you should buy one.

What is NPU?

NPU is an acronym for Neural Processing Unit. NPUs are designed to perform mathematical functions related to neural networks/machine learning/AI tasks. Although they can be stand-alone chips, they are increasingly being integrated directly into a system-on-chip (SoC) alongside the more familiar CPU and GPU components.

NPUs come in a variety of shapes and sizes, and are often called slightly differently depending on the chip designer. You’ll already find different models scattered across the smartphone landscape. Qualcomm has Hexagon in its Snapdragon processors, Google has its TPUs for both the cloud and its mobile Tensor chips, and Samsung has its own Exynos implementation.

The idea is now spreading to laptops and computers. For example, there’s the Neural Engine in the latest Apple M4, Qualcomm’s Hexagon features in the Snapdragon X Elite platform, and AMD and Intel have started integrating NPUs into their latest chipsets. While not quite the same, NVIDIA GPUs blur the lines given their impressive number crunching abilities. NPUs are increasingly common everywhere.

As mentioned, NPUs are specifically designed to handle machine learning workloads (along with other heavy math tasks). In layman’s terms, the NPU is very useful, perhaps even an essential component for running AI on a device rather than in the cloud. As you’ve no doubt noticed, AI seems to be everywhere these days, and building support directly into products is a key step in that journey.

Much of today’s AI processing takes place in the cloud, but this is not ideal for several reasons. First is latency and network requirements; you don’t have access to tools when you’re offline or you may have to wait a long time for processing during peak hours. Sending data over the Internet is also less secure, which is a very important factor when using AI that has access to your personal information, such as Microsoft’s Recall.

Simply put, it’s preferable to work on the device. However, AI tasks are very computationally intensive and do not perform well on traditional hardware. You may have noticed this if you tried to generate images using Stable Diffusion on your laptop. It can be painfully slow for more complex tasks, although the CPUs can handle a number of “simpler” AI tasks quite well.

The solution is to adopt dedicated hardware to accelerate these advanced tasks. You can read more about what NPUs do later in this article, but the TLDR is that they perform AI tasks faster and more efficiently than your CPU can do on its own. Their performance is often quoted in trillions of operations per second (TOPS), but that’s not a particularly useful metric because it doesn’t tell you exactly what each operation is doing. Instead, it’s often better to look for numbers that tell you how fast token processing is needed for large models.

Speaking of TOPS, NPUs for smartphones and early laptops are rated in the tens of TOPS. Broadly speaking, this means they can speed up basic AI tasks, such as detecting objects from the camera to apply bokeh blur or text summarization. If you want to run a large language model or use generative AI to quickly produce media, you’ll want a more powerful accelerator/GPU in the hundreds or thousands of TOPS range.

A neural processor is quite different from a central processor because of the type of workload it is designed for. The typical processor in your laptop or smartphone is fairly general-purpose to take care of a wide range of applications, supporting a wide set of instructions (functions it can perform), various ways of caching and calling functions (to speed up repetitive loops) and large out-of-order execution windows (so they can keep doing things instead of waiting).

However, machine learning workloads are different and don’t need as much flexibility. For starters, they are much more math-heavy, often requiring repetitive computationally expensive instructions like matrix multiplication and very fast access to large arrays of memory. They also often work with unusual data formats, such as sixteen-, eight-, or even four-bit integers. In comparison, your typical CPU is built around 64-bit integer and floating-point math (often with extra instructions thrown in).

Building an NPU dedicated to massively parallel computation of these specific functions results in faster performance and less energy wasted on idle functions that are not useful for the task at hand. However, not all NPUs are created equal. Even beyond their pure number processing capabilities, they can be built to support different integer types and operations, meaning that some NPUs perform better on certain models. Some smartphone NPUs, for example, work with INT8 or even INT4 formats to save power, but you’ll get better accuracy from a more advanced but power-hungry FP16 model. If you need really advanced computing, dedicated GPUs and external accelerators are still more powerful and diverse in format than integrated NPUs.

As a backup, CPUs can perform machine learning tasks, but are often much slower. Modern processors from Arm, Apple, Intel, and AMD support the necessary math instructions and some of the smaller quantization levels. Their bottleneck is often exactly how many of these functions they can run in parallel and how fast they can move data in and out of memory, which NPUs are specifically designed to do.

While far from essential, especially if you don’t care about the AI ​​trend, NPUs are necessary for some of the latest features you’ll find in the mobile and PC space.

Microsoft’s Copilot Plus, for example, specifies an NPU with 40TOPS performance as the minimum requirement you’ll need to use Windows Recall. Unfortunately, Intel’s Meteor Lake and AMD’s Ryzen 8000 chips found in current laptops (at the time of writing) do not meet this requirement. But AMD’s recently announced Stix Point Ryzen chips are compatible. You won’t have to wait long for an x64 alternative to the Arm-based Snapdragon X Elite laptops, as Stix Point-powered laptops are expected in the first half of 2024.

Popular desktop-grade tools like Audacity, DaVinci Resolve, Zoom, and many others are increasingly experimenting with more demanding on-device AI capabilities. While not essential for core workloads, these features are becoming increasingly popular and AI capabilities should factor into your next purchase if you regularly use these tools.

When it comes to smartphones, features and capabilities vary a little more widely depending on the brand. For example, Samsung’s Galaxy AI only works on its high-powered Galaxy S flagship phones. It hasn’t brought features like a chat assistant or translator to the affordable Galaxy A55, probably because it lacks the necessary processing power. However, some of Samsung’s features also work in the cloud, but they probably aren’t funded with more affordable purchases. Speaking of which, Google is equally as consistent in terms of features. You’ll find the best of Google’s AI goodies in the Pixel 8 Pro, like Video Boost — though the Pixel 8 and even the affordable 8a run many of the same AI tools.

After all, AI is here and NPUs are the key to enjoying device features that can’t run on older hardware. That said, we’re still in the early days of AI workloads, especially in the laptop space. Software requirements and hardware capabilities will grow in the coming years. In that sense, waiting for the dust to settle before jumping in won’t hurt.