In a recent conversation with Stevie Bathiche, a Technical Fellow at Microsoft, we discussed the potential of the Neural Processing Unit (NPU) and its role in powering the future of computing. Bathiche’s insights highlighted the NPU’s significance as an AI coprocessor due to its efficiency and ability to enable always-on, always-computing functionalities that power computers to be always learning, sensing, and computing on our behalf.  Something that was not possible before the NPU, the newest chip on the block. 

The Evolution of Processing Units

Bathiche drew an interesting analogy between the introduction of math coprocessors in the 1990s and the development of modern Neural Processing Units (NPUs). In the past, math coprocessors were designed to alleviate the burden on the main CPU by handling complex mathematical calculations. Similarly, NPUs are now being developed to efficiently process the intricate data structures and mathematical operations required by artificial intelligence algorithms.

At the heart of AI lie tensors, which are essentially arrays of vectors, with vectors being arrays of scalars. Processing these complex data structures demands specialized hardware to ensure optimal performance. Just as math coprocessors were introduced to accelerate complex calculations, NPUs are being designed to tackle the unique challenges posed by AI workloads.

By offloading AI-specific tasks to dedicated hardware, NPUs allow the main CPU to focus on general-purpose computing tasks, thereby improving overall system efficiency and performance. This parallel between math coprocessors and NPUs highlights the ongoing efforts to develop specialized hardware that can keep pace with the rapidly evolving field of artificial intelligence.

“NPUs are AI coprocessors built to manage the structure of tensors and the math around AI. This specialization allows for highly efficient AI workload acceleration, similar to how GPUs revolutionized graphics rendering.” – Stevie Bathiche

Efficiency and Performance Gains

The primary advantage of NPUs lies in their efficiency and performance. While CPUs and GPUs can handle AI workloads, they are not optimized for the specific demands of these tasks. NPUs, on the other hand, are purpose-built, resulting in significant gains in speed and power efficiency. Bathiche illustrated this with a comparison: an NPU can handle AI workloads 30 to 40 times faster than a CPU while consuming significantly less power. This substantial improvement can be attributed to the specialized architecture of NPUs, which feature circuits and data paths designed specifically for processing tensors and performing complex mathematical operations.

The power efficiency of NPUs is particularly crucial in scenarios where AI is deployed on edge devices or in power-constrained environments. The ability to perform AI tasks with significantly lower power consumption enables the integration of AI capabilities into a wider range of devices, from smartphones and smart home appliances to industrial sensors and autonomous vehicles. As AI continues to permeate our daily lives and transform businesses, the role of specialized hardware like NPUs will become increasingly important in ensuring the efficient and effective deployment of AI solutions.

“An NPU has a TDP (thermal design power) of around 4 watts, compared to the 20-30 watts of a CPU. This makes the NPU not just faster, but also five to seven times more power-efficient.” – Stevie Bathiche

The Always-On, Always-Computing Paradigm

One of the more interesting aspects of NPUs is their potential to enable always-on, always-computing capabilities. Bathiche emphasized that this constant computation is essential for unlocking new software innovations and enhancing user experiences. For instance, Microsoft’s Recall project, which provides users with a form of photographic memory by indexing and semantically compressing information viewed on their screens, relies on the NPU to run continuously without draining battery life. This highlights a key architectural insight of NPUs given their ability to run in the background at low-wattage while also processing massively complex computational workloads.  Compared to something like a CPU, or even a GPU, which is designed to run a workload as fast as possible and then try to get to a zero state to preserve battery life.

The ability to perform continuous computation without significantly impacting power consumption opens up a wide range of possibilities for AI applications. Always-on AI can enable proactive and context-aware experiences, such as intelligent virtual assistants that can anticipate user needs based on real-time data analysis. This constant computation also facilitates the development of more responsive and intuitive user interfaces, as well as the implementation of advanced security features like continuous authentication based on biometric data.

“Without an NPU, running such persistent inference workloads would drastically reduce battery life. But with the NPU, devices can maintain long battery life while performing complex AI tasks in the background.” – Stevie Bathiche

A New Era of Software Development

Bathiche believes we are on the cusp of a revolutionary era in software development, driven by the capabilities unlocked by NPUs. This new hardware allows developers to create more complex, AI-driven applications that were previously impossible due to resource constraints.

“Every major leap in computing has been accompanied by new software that abstracts and leverages the increased compute power. Today is no different. The NPU enables a new class of applications and features, such as AI-driven productivity tools, advanced image processing, and real-time contextual assistance.” – Stevie Bathiche

Microsoft’s Vision for the Future

Microsoft’s journey with NPUs and their integration into devices like the latest Surface models underscores the company’s commitment to harnessing this technology for the benefit of users and developers alike. Bathiche shared that Microsoft has successfully demonstrated running multiple AI workloads simultaneously on a Surface device without compromising performance, showcasing the NPU’s potential.

“We ran all AI models, studio effects, background blur, eye contact, audio denoiser, image generation, and a small language model—all at once—without any lag. This is the magic of shuttling the right workload to the right processor efficiently.” – Stevie Bathiche

Conclusion

Stevie Bathiche’s insights underscore the transformative potential of NPUs in the tech industry. By enabling efficient, always-on AI computation, NPUs are poised to redefine the capabilities and performance of devices across various domains. As Bathiche puts it, “The developer, company, or platform that figures out how to leverage this always-on computing power will make a significant impact on the future of technology.”

In these exciting times for software development, as Bathiche encourages, software developers should seize the opportunity to harness the capabilities of NPUs and unlock new potential in their applications.

View the full interview below.

 

Join the newsletter and stay up to date