REPORT: TOPS of the Morning to You

May 20, 2024 / Ben Bajarin and Max Weinbach

Table of Contents

Given there is a new standard in measuring NPU performance capabilities, it is critical we have a better understanding of the relationship between NPU TOPS and the relationship of TOPS as a metric and AI performance. While we completely agree, TOPS is a difficult metric to measure real-world AI performance, we think of it in terms of CPU GHz measurements in terms of knowing potential raw performance of the NPU. Below, is a brief explainer on some of the ways we expect TOPS to be leveraged, and communicated and the role TOPS will likely play in understanding absolute AI performance potential. 

Direct Proportionality: Generally, a higher TOPS rating indicates better performance. An NPU capable of delivering a higher number of TOPS can process more neural network operations per second, leading to faster execution of AI tasks. For example, an NPU with 10 TOPS will theoretically be able to process AI tasks about twice as fast as an NPU with 5 TOPS, assuming all other factors are equal.

Benchmark for AI Performance: TOPS has become a standard benchmark for measuring the performance of AI accelerators like NPUs. It allows for a direct comparison of the raw computational power of different NPUs. However, it’s important to note that TOPS is a theoretical peak performance metric and doesn’t always directly translate to real-world performance.

Influence on Model Complexity: A higher TOPS rating allows an NPU to handle more complex neural network models. As AI models become more sophisticated with deeper layers and more parameters, they require more computational power. An NPU with a higher TOPS rating will be better equipped to process these complex models efficiently.

Power Efficiency Consideration: While a higher TOPS rating generally indicates better performance, it’s also important to consider the power efficiency of the NPU. The TOPS/Watt metric is often used to evaluate how efficiently an NPU uses its power to deliver its performance. A more power-efficient NPU can deliver better performance per watt, which is particularly important for battery-powered devices like smartphones, tablets, and laptops.

Real-World Performance: While TOPS provides a good indication of an NPU’s potential, real-world performance can vary based on factors such as the efficiency of the software stack, the optimization of the AI models for the specific NPU architecture, and the thermal constraints of the device. Therefore, TOPS should be considered alongside other factors when evaluating the actual performance of an NPU in practical scenarios. 

Next ❯ Copilot+ PCs and a New Category of AI PC

Join the newsletter and stay up to date

Trusted by 80% of the top 10 Fortune 500 technology companies