Day One of the NPU For Software Developers
Table of Contents
Over the course of this series, we have been exploring ways developers have been utilizing the NPU, and taking advantage of its dedicated AI processing capabilities. While we still have more developer stories to tell, one clear insight has stood out from many of our discussions with a gamut of software developers.
When it comes to software development, a timeless phrase in the industry is for software developers there is never enough compute. Essentially, give software developers more computing capabilities and they will always utilize it. This has typically been true of the CPU, GPU, and now we expect this to be true of the NPU.
While the NPU has been around for many years, we are still in the early days of exploring the software development potential. A survey of the landscape of NPU capabilities on the market, and coming to market over the next 6-8 months, looks to average around 44 TOPS of NPU capable compute. That is the estimated average of NPU TOPS coming from Apple, Qualcomm, AMD, and Intel. So from a software development standpoint, specifically from an NPU capabilities standpoint, all vendors are in the same ballpark. This is why I say not only is it early days, but there is no clear NPU leader, which means there is a relatively level playing field for software development that utilizes this unique AI core.
This is a point that numerous software developers have made in our interviews as they recognize the level playing field today but have consistently said that they are extremely interested to see how the capabilities of NPUs evolve because they believe as NPUs become more capable, it will only open more opportunities for them to push the limits of their software experiences and add even more value and new features for their customers.
We expect the NPU to be one of the biggest beneficiaries of all cores on the SoC of gen on gen performance increases. It is not out of the question that 75-100 NPU TOPS is in our foreseeable future, which will undoubtedly unlock even more AI experiences on device and empower developers to create even more compelling software infused with AI.
Not All NPUs May be Equal
Where this gets extremely interesting in our continued exercise of evaluating the NPUs role in the future of computing, is how Apple, Qualcomm, AMD, and Intel all have entirely different IP and architectural approaches to their NPUs. Unlike a standard microarchitecture like Arm, or x86, there is no standard for NPUs. Every NPU recipe is unique and entirely different.
Where this excites us from an analysis standpoint, is the ability to truly evaluate each vendors NPU in a range of different benchmarks. However, the lack of standardization makes this analysis even more difficult. That said, when it comes to doing genuine architectural analysis, and one where we can truly appreciate the design variances from skilled semiconductor engineers, the NPU opens the door to some of the most interesting technical analysis on architectural design in some time.
This variance in NPU design brings up an interesting challenge for software developers. While tools exist to optimize their software specifically for different vendor NPUs, it is unlikely anyone other than the largest Tier 1 software vendors take the time to optimize their software for different vendors NPU.
In different venues, Sam Altman, CEO of OpenAI, has stated his conviction that this may be one of the most exciting times to be a software developer. Having talked to numerous developers now about the software opportunities the NPU opens up, I think of all the times to be a software developer, this is easily the most exciting and interesting time to write software. We are in the early stages of this AI software cycle but I’m confident that we have many years ahead of continual advancements in compute capability that will fuel an entire new generation of apps infused with AI.