NPUs (Neural Processing Units) are rapidly establishing themselves as the optimal platform for executing a diverse array of AI workloads, especially those requiring prolonged periods of inference. The unique architecture of NPUs, tailored specifically for neural network operations, guarantees efficient and high-performance processing. This architecture is adept at managing parallel computations, employing low precision arithmetic, and incorporating advanced power management techniques, making NPUs exceptionally well-suited for sustained AI inference tasks.

The integration of on-chip memory in NPUs further enhances their efficiency and power consumption. This memory integration allows for quicker access to data and reduces the need for data transfer between different components, thereby minimizing latency and energy usage. Optimized workloads that are specifically designed for NPU architecture also contribute to their superior performance and efficiency.

Looking forward, we plan to expand our NPU testing by including more specific use cases from Microsoft and third-party developers. This expansion will help in understanding the practical applications and benefits of NPUs in real-world scenarios. Additionally, we will continue to develop a series of benchmarks and analyses that will highlight the critical role of NPUs and their symbiotic relationship with CPUs and GPUs in handling AI workloads. These benchmarks will provide valuable insights into how NPUs, CPUs, and GPUs can work together to deliver optimal performance for various AI tasks.

In conclusion, the specialized capabilities of NPUs make them a cornerstone technology for AI inference, promising continued advancements and integration into diverse applications. Our ongoing testing and benchmarking efforts will provide deeper insights into their performance and efficiency.

Table of Contents

• The New Chip on the Block 
• What Makes the NPU Unique
• TOPS of the Morning to You
• Copilot+ PCs and a New Category of AI PC
• Developer Spotlight: Cephable
• The NPU Wattage Advantage
• Conclusions and What’s Next

*This white paper was commissioned by Microsoft. The insights and analyses provided are based on research and data obtained through collaboration with Microsoft and third-party developers. The goal of this paper is to present an unbiased examination of NPUs and their role in AI workloads. While Microsoft has provided support for this research, the findings and conclusions drawn in this document are those of the authors and do not necessarily reflect the views of Microsoft.

Join the newsletter and stay up to date