Arm in the Data Center: The Dawn of a New Era

October 31, 2024 / Ben Bajarin

Introduction: 2024 – The Tipping Point for Arm’s Data Center Momentum

As we close out 2024, the semiconductor industry is witnessing a significant shift in the data center market. AI is the most important technology platform shift we have seen in a decade, and perhaps longer.  This fundamental shift is causing the entire technology industry to begin to rethink its infrastructure and at the heart of that infrastructure is the silicon which powers this new AI era of computing. The impact of this reality is that the vast majority, if not all, of data center infrastructure needs to be upgraded, and in many cases redesigned. Computing backend infrastructure is moving away from a general-purpose platform to more specialized solutions built with AI workloads in mind. With any new platform shift, it opens the opportunity for incumbents to be disrupted as customers will consider any and all technology vendors. Moats that once existed may no longer be as wide or deep.  It is within this context we look at the ways Arm-based processors are gaining traction in the data center. While Arm has been touted as a potential contender for years, several factors suggest that this time is different, with 2024 potentially representing the “tipping point” for Arm’s adoption in the data center.

Market Growth and Proof Points:

While much of the data center semiconductor content growth is coming from GPUs and AI accelerators, there is still growth to be had in CPUs. What is interesting, however, is during this time of infrastructure turnover the market seems to be segmented into AI servers and general-purpose (non-AI servers).  Each category still needs a host CPU, however, the ratio of CPU to GPU or accelerator in AI servers may very well drop from general-purpose servers which have more CPUs to GPUs.  This is one reason the data center CPU market is only projected to grow at 3% CAGR for the next four years. 

However, the bright spot in CPU datacenter growth is Arm-based CPUs which are poised to outgrow the market. 

The Rise of Arm in the Server Market

While the overall server market shows steady growth, Arm-based servers are experiencing a meteoric rise. Forecasts from Wells Fargo project:

  • Arm server shipments are forecast to skyrocket from 7.1% of shipments in 2023 to 22% of shipments by 2027.
  • This represents a CAGR of 40% for Arm-based server shipments.
  • In terms of market share, Arm-based servers are expected to grow from just 3.9% of total server shipments in 2022 to 17.9% by 2027. To further cement this point, Arm is projected to outgrow the market by 4x. 

(*Note: all projects and forecasts above are from third-party sources, our own research and analysis, and are not representative of Arm’s internal forecasts or models)

The increased adoption and prioritization of custom silicon by the hyperscalers is another key reason for Arm’s growing adoption. Just to name a few initiatives. 

  • Amazon’s Graviton Success: AWS has reported significant benefits with their Arm-based Graviton processors now 4th generation:
    • 40% better price-performance compared to x86 alternative
    • Up to 60% better energy efficiency
  • Hyperscaler Adoption: Microsoft Azure and Google Cloud have announced custom Arm-based chips for both their own data center applications and as part of their respective cloud service offerings. 

“Arm engages at a fundamentally deeper level with our partners than any other compute vendor in the industry.  We work together to enable a unique path to the custom compute that today’s computing infrastructure needs. This collaboration directly influences how Arm evolves its architecture – and how we implement the micro-architecture in our IP products.” – Dermot O’Driscoll, VP of Infrastructure Product Solutions, Arm

  1. NVIDIA’s Grace CPU: NVIDIA has long been an Arm customer but in recent years their strategy to embrace Arm much more extensively has become more clear and in many ways more strategic to their ambitions in the data center and beyond.  NVIDIA’s investment in Arm with their Grace CPU as the preferred host CPU to their current and future GPUs is further validation of Arm’s role in the data center. 

Next, let’s look at some of the reasons driving Arm’s growth in the data center. 

Evolution of the Workload – From Chips to Systems

The semiconductor industry is undergoing a fundamental shift from chip-centric to systems-centric computing, driven primarily by AI workload demands. While traditional approaches focused on improving individual chips, success now requires optimizing entire computing systems – from processors and memory to networking and software. Even minor system-level improvements can create substantial value, particularly in large-scale AI deployments.

Arm is well-positioned for this transition through its compute subsystems approach. Rather than just licensing processor cores, Arm provides complete computing solutions that partners can customize with accelerators, security features, and interconnect technologies. Their flexible model enables partners to optimize systems for specific AI workloads while maintaining software compatibility.

Arm’s established success in mobile computing, combined with their extensive partner ecosystem and system-level expertise, gives it a strong foundation in the AI infrastructure market. As companies seek customized AI solutions, Arm’s approach offers the scalability and flexibility needed while preserving a unified software ecosystem.

Performance and Efficiency:

The perception of Arm-based processors has undergone a dramatic shift in the server market, evolving from an alternative architecture to a performance leader. Today’s Arm implementations combine superior per-core performance with exceptional power efficiency, challenging traditional assumptions about server processor design. This technical approach to microarchitecture, coupled with inherent flexibility, has captured the attention of major cloud providers and data center operators who seek both performance gains and improved operational efficiency.

  • Performance: once derided as “underpowered”, Arm’s per-core performance now exceeds that of leading x86 SKUs on many applications. And because the Arm architecture foregoes simultaneous multi-threading (aka, “hyper-threading”), Arm’s performance scales better with added core count.
  • Power Efficiency: Arm’s RISC architecture and CPU implementations, originally designed for mobile devices, offer superior performance-per-watt metrics compared to x86 alternatives. For servers, the Arm Neoverse CPU IP was designed at the microarchitectural level for efficiency. This inherent advantage has become increasingly important as data centers grapple with rising energy costs and the need to reduce their environmental footprint.
  • Scalability: Arm’s architecture allows for efficient scaling across a wide range of performance levels, from edge devices to high-performance servers. 
  • Customization: The ability to tailor Arm-based designs for specific workloads enables optimized performance for cloud-native applications. 
  • Hyperscaler Focus: Major cloud providers are aggressively pursuing energy-efficient solutions, with Amazon leading the charge through its Graviton processors.

Flexibility and Pace of Innovation – Arm’s Key Differentiator

The data center landscape is characterized by relentless innovation, with new technologies and architectures emerging constantly. In this dynamic environment, the ability to rapidly adapt and deliver cutting-edge solutions is essential for success. A great example of this is Amazon’s AWS has been able to launch 4 different generations of Arm-based processors in 5 years. Arm’s business model, centered around its IP licensing model and the development of flexible and adaptable architectures, positions it well to thrive in this fast-paced environment. Arm’s introduction of pre-configured, pre-validated “compute subsystems” (CSS), furthers this advantage. 

Custom Silicon at Hyperscalers:

One of the biggest trends benefitting Arm is the continued investment by many of the world’s top technology companies to custom silicon. This is a trend our firm has predicted and has been following closely ever since Apple went all in on custom silicon. 

The driving force for companies to continue to invest and scale their own custom silicon solutions is to expand their differentiation. The challenge with merchant silicon is your competitors can also buy the latest and greatest silicon solutions and thus your differentiation is limited.  

Alan Kay famously said, “those who are passionate about software should also be passionate about hardware.” While that quote has been interpreted many different ways and is noted as a primary reason Apple has always strategically been deeply vertically integrated, the quote simply means a deep understanding of hardware can significantly enhance software development and innovation and lead to more specialized and highly differentiated solutions.  We believe that quote applies today and must be interpreted as including semiconductor hardware, and we argue that any company that seeks to create and control a platform will invest in more custom silicon strategies.  In the hyperscaler market, we see clear examples of this trend:

  1. Amazon AWS Graviton: Now in its fourth generation, demonstrating Amazon’s commitment to Arm-based custom silicon
  2. Microsoft Azure Cobalt: In its first generation, Microsoft created Arm-based chips named Cobalt for cloud and edge computing scenarios
  3. Google Cloud Axion: Also in its first generation Google has developed  Arm-based designs named Axion to complement existing TPU offerings.

Arm’s IP and compute subsystems (CSS) are especially well-suited to enabling the custom silicon trend due to its unique ability to lower the cost of design while simultaneously enabling chip designers to focus their resources on differentiation rather than commodity components.

  • Cost: The cost of developing advanced SoCs at leading-edge nodes has increased dramatically, presenting a challenge for companies seeking to develop custom chips. Arm addresses this challenge by providing a more cost-effective way to develop custom silicon through its licensing model, which allows companies to leverage Arm’s existing processor architectures, IP blocks, and compute subsystems (CSS), rather than developing everything from scratch. This is particularly beneficial for companies that may not have the resources or expertise to design their own processors in-house.
  • Customization and Time to Market: Arm’s Compute Subsystems, such as the Neoverse CSS N2, provide pre-validated and pre-integrated configurations of Arm’s IP, further reducing design complexity and time to market. These subsystems include a range of pre-configured options for core count, frequency, and other features, allowing customers to quickly assemble a customized solution that meets their specific needs. This approach significantly accelerates the design process, allowing companies to bring their custom silicon solutions to market faster and with lower risk. For example, one Arm partner was able to go from project inception to booting Linux on 128 CPUs in just 13 months using CSS.
  • Focus on Differentiation: By leveraging Arm’s IP and CSS, companies can focus their engineering resources on developing the differentiating features of their custom silicon, such as specialized accelerators or customized I/O, rather than spending time and resources on commodity components like CPUs or interconnects. This is critical for companies seeking to create unique silicon solutions that provide a competitive advantage.
  • Software Ecosystem: Arm also benefits from a robust software ecosystem, built up over decades of experience in mobile and embedded computing. This mature ecosystem makes it easier for companies to develop and deploy software on Arm-based custom silicon, further lowering the barrier to entry for companies seeking to adopt custom silicon solutions.
  • Addressing the Needs of a Changing Datacenter Landscape: The rise of hyper-scale cloud providers and the increasing demand for specialized workloads like AI inference are driving the need for more customized and efficient data center solutions. Arm’s technology, with its focus on low power consumption, scalability, and customization, is well-positioned to meet these evolving requirements. For example, the company’s Neoverse platform, specifically designed for data center and networking applications, offers a compelling alternative to traditional x86-based solutions for many workloads.
  • The Rise of AI and Specialized Workloads: The increasing use of AI, especially generative AI, demands immense processing power that goes beyond the capabilities of traditional general-purpose CPUs. These specialized workloads benefit from purpose-built hardware like AI accelerators, which are driving the adoption of custom silicon.
  • The Slowing of Moore’s Law: As Moore’s Law, which predicts the doubling of transistors on a microchip every two years, slows down, companies are seeking alternative ways to increase performance. Custom silicon, with its ability to be tailored to specific workloads, offers a way to continue advancing performance even as traditional scaling methods become less effective.
  • The Growing Importance of Energy Efficiency: Data centers consume a significant amount of energy. As these facilities grow larger to accommodate the demands of AI and other data-intensive applications, energy consumption is becoming an increasingly important concern. Arm-based processors, known for their energy efficiency, are well-suited for this environment. This focus on efficiency also extends to optimizing power delivery networks (PDNs) within data centers.
  • The Need for Data Sovereignty: Companies are increasingly concerned about data privacy and security. Custom silicon, particularly for edge inference applications, allows companies to process data locally, potentially reducing the need to send sensitive information to the cloud.

The rise of custom silicon, particularly through Arm’s technology and licensing model, represents a significant shift in the data center landscape. Major cloud providers like Amazon, Microsoft, and Google are increasingly adopting Arm-based custom chips to address modern computing challenges. Arm’s approach, combining cost-effective licensing, pre-validated compute subsystems, and a mature software ecosystem, enables companies to develop specialized solutions while focusing on differentiation. This strategy is particularly timely given current industry challenges, including the slowing of Moore’s Law, increasing energy efficiency demands, and the rise of AI workloads. By providing a balance of accessibility, customization, and efficiency, Arm is well-positioned to support the industry’s evolution toward more specialized and efficient computing solutions.


Next: Arm’s Software Ecosystem: Growing Breadth of Support for Critical Workloads

Join the newsletter and stay up to date

Trusted by 80% of the top 10 Fortune 500 technology companies