AMD Series · Part 3 of 3
Compute choice, fungibility, and the cost of correctly completed work
As companies put AI into everyday work, we expect attention to shift toward everything required to complete a task. An agent may use a model to interpret a request, retrieve company information, call an application, and then check the result. Those steps use different parts of the system, so the cost depends on how well the software assigns the work and how much time is spent waiting between steps.
We see a broader role for AMD here. EPYC, Instinct, networking, and ROCm each address a different part of that work, and software and system partners help customers put them into service. The question we are most interested in is how those investments change the cost of running an application over time. The customer needs a system that handles today's work and can change models or add capacity without the application being rebuilt each time.
Our thesis that the future of compute is fungible starts with that need for choice. By fungible, we mean that operators and customers can direct a workload to whichever compute serves it best without rebuilding the application around each change. Some of that is already here. An application can use a model through an API without choosing the processor that serves each request, and routing services already let customers select among providers behind a common interface. That gives AMD an opportunity to supply the infrastructure beneath services customers already use, as well as the systems they buy directly. The provider still chooses and prepares the hardware, but much of that work no longer has to be repeated by every application developer. [8]
The assumptions behind our view
Our view rests on four assumptions. We expect enterprises to use a mix of models and tasks with different cost and response-time needs, and those differences give customers reasons to choose where the work runs, although the savings will depend on the application. We also assume that common APIs, which already support choice between model endpoints, keep widening that choice. How much work can actually move between endpoints depends on model availability, application features, data rules, and the provider's supported systems, and it does not require different GPUs to share one model instance.
The economics further assume that customers have enough work to justify the capacity they buy. Low utilization or repeated integration work can outweigh a lower hardware price, and an application still has to deliver value to the business. Finally, AMD's software and partners give customers more than one way to use its hardware, and we expect the strongest economic benefit where a supported configuration meets the application's quality and response-time needs while reducing the cost or power required to serve it.
Software gives customers more ways to choose compute
Customers can use AMD through a hosted inference service, a new internal application, or infrastructure they operate themselves. A common model API makes the first route relatively simple for the application team, because the service provider handles the hardware and serving software. Buying equipment gives the customer more control over data and capacity, with more responsibility for operating it. Both routes widen the set of workloads AMD can serve without asking customers to replace everything they already run.
The application still has to return an acceptable answer, handle tool calls correctly, and meet the company's data and security rules. Those are normal requirements for deploying AI on any platform. A provider or system partner that has already done the hardware integration reduces the work left for the customer, which means the same processor can be easier to adopt through a supported service than through a system the customer assembles itself.
We see AMD's work on integrated systems such as Helios and its open software interfaces as two parts of the same effort. Helios provides a rack design that partners turn into systems for training and inference. Common interfaces help those systems fit the applications customers already run, and other AMD components can be bought separately for the surrounding infrastructure. The customer gets to decide how much integration to buy from a partner and how much to control itself. [1, 9]
The combination works because the customer's existing software has value. A new inference endpoint can serve an application without changing the databases or business services around it, and an EPYC server can support those services regardless of which accelerator serves the model. AMD can therefore compete at more than one point in the workflow, and each system should be judged on the performance and cost of the work it takes on.
Software makes the hardware available to applications
It helps to separate the software layers (Exhibit 1). ROCm is AMD's GPU software platform, including runtimes, libraries, and development tools; CUDA fills a comparable role in NVIDIA's ecosystem, although their capabilities and interfaces are not identical. Many developers work above those layers, through frameworks such as PyTorch and serving engines such as vLLM. The framework expresses the model's operations, the serving engine manages requests and execution, and hardware-specific software runs the work on the GPU. Supporting familiar frameworks lets AMD reach developers without asking all of them to write directly against a new low-level platform. [9]
AMD Inference Microservices, or AIMs, package model serving into containers with profiles for supported hardware and models. For language models, AIMs expose an OpenAI-compatible API, as NVIDIA NIM also does, so an application using the shared API features may need relatively few changes to use another endpoint. The model name, authentication, tool behavior, and any service-specific features still have to match what the application expects. A common interface reduces migration work without making every endpoint identical. [2, 10]
AMD's enterprise software stack brings container deployment and serving tools together, with Kubernetes managing where containers run and the resources assigned to them. AMD also works with platform partners including Red Hat, SUSE, Canonical, VMware, and Nutanix so that customers can use familiar operating, virtualization, and container environments. Support depends on the specific product and configuration, and not every partner offers the same stack. For an enterprise, the benefit is being able to add AI to the environment its teams already know how to operate. [3, 11]
Cloud operators and enterprises will use that support differently. A provider with its own software team will choose and tune the infrastructure beneath an API. An enterprise is more likely to want a partner to supply the supported system and operating environment. AMD can serve both without pushing the hardware work onto the application developer. At that point openness shows up in deployment time and support effort, which is where customers feel it, and not only in the terms of a software license.
Model routing and hardware placement solve different problems
Choosing the model is one decision. A routine classification may be handled by a much smaller model than a complex planning task, so sending both to the same expensive service wastes resources when the smaller model already meets the need. Tools such as LiteLLM route and balance requests among endpoints, and vLLM documents a Semantic Router integration that selects models according to the request. That integration is still labeled a preview, but it shows model selection and serving starting to converge. [12]
Deciding where to run that model involves a different set of questions (Exhibit 2). A server that already has the model loaded and the relevant context cached may serve a request more efficiently than an idle machine that has to load the weights and repeat earlier work. The software has to account for that preparation along with the queue, the available memory, and the expected response time. Looking only for the least busy GPU misses part of the cost.
A business can use that approach to send routine, permitted work to a privately hosted model and reserve a frontier service for the tasks that need it. The router has to account for answer quality and data policy as well as price. A cheaper response that triggers retries or human correction may cost more across the full task, so the savings have to be measured after the work is accepted rather than at the first model call.
This is also why our compute-fungibility thesis distinguishes between routing requests and splitting a model across different hardware. A router selects an endpoint, and the provider or serving system determines which devices execute the model behind it. Customers get real choice between services without mixing accelerator vendors within a single model instance. The more closely the hardware has to cooperate on the same model, the more its memory, communication, and runtime support decide the outcome.
Memory and communication set the useful capacity
The work changes during inference itself (Exhibit 3). Processing a prompt, called prefill, leans heavily on compute. Generating the response, called decode, puts more pressure on memory bandwidth. Meta's published inference work describes those differences and the potential to use different hardware for the two stages. The balance depends on the model, the context length, and how many requests are processed together, so a single hardware ratio will not suit every service. [4]
Separating those stages means the information built during prefill has to reach the system generating the answer. The gain from using hardware suited to each stage has to cover the transfer time, the network traffic, and the added scheduling work. We would expect that tradeoff to look different for a large, steady service than for a smaller deployment that already fits on one server.
Memory capacity also governs how many requests a system can serve at once. Alongside the model weights, transformer serving uses a key-value cache to retain information from earlier tokens, and longer contexts and more concurrent sessions consume more of that memory. Reusing a shared prompt prefix avoids some repeated processing, and lower-precision representations reduce memory use when they preserve the required quality. AMD's ROCm optimization guidance addresses these serving choices, which need to be tuned to the workload. [5] Exhibit 4 shows why agents feel this more than chat applications do. Agentic inputs carry tool results and company context, so they tend to run long, and at long context lengths the cache rather than the weights sets the ceiling on concurrent sessions.
For Instinct, memory capacity and bandwidth affect how much work the customer can serve from the GPUs it buys. More memory may allow a model to fit on fewer GPUs or leave room for additional sessions, although that benefit depends on what else limits the system, including compute throughput and queues. Sharing or partitioning a GPU also helps smaller workloads use capacity, provided the available isolation and resource limits meet the customer's requirements.
A mixture-of-experts model adds communication between the GPUs hosting its experts as tokens are processed, and Meta describes this exchange as a source of inference latency. That is one reason faster links need to be considered alongside how work and experts are distributed. For AMD, networking is tied directly to the cost of serving the model, because time spent waiting on data is time the accelerator is not doing the work the customer bought it to do. [4]
Agent execution needs its own operating controls
Our Agentic CPU Guidebook follows the surrounding work across accelerator hosts, separate tool environments, and existing business services. A coding agent may wait for a build or test before making its next model call, so faster execution shortens that dependent step, while more cores help when many independent jobs are ready at once. EPYC gives AMD a way to serve both requirements across these pools, including work generated by models running on another supplier's accelerators. [13]
The number of hardware threads does not tell us how many agents a server can support. Some environments spend most of their time waiting. Others run builds or tests that use several cores. Sharing execution capacity or suspending inactive environments changes how much equipment the operator needs, and memory may limit the number of active jobs before the CPU is fully used. We would size the system from the work, its memory needs, and the required response time, rather than treating every agent as a permanently occupied core.
Existing x86 applications also give AMD a familiar base for the business services agents call, and enterprises can keep much of their software, identity management, and operating practice as they add agent workflows. They still need permissions for each tool, a record of actions, and controls on resource use. VM isolation, confidential computing, and application permissions each solve a different problem. Offloading network services can free CPU time, but it does not decide whether an agent should be allowed to change a business record.
On premises and cloud can serve the same business
We expect enterprises to decide where AI runs according to the work they need done. Steady inference on sensitive company data is a reason to consider private capacity. Unpredictable demand, or the need for a particular model, points toward a cloud service. Owning equipment in the company's facility or in colocation carries different responsibilities from renting access to a model in the cloud, even when both serve the same business.
Private equipment has to be paid for and supported even when demand falls. The customer needs to include deployment, power, ongoing support, and capacity held for failures or peaks in its cost. A cloud service charges for the work sent to it, plus data movement costs where they apply and the effort of maintaining the connection. How steadily the company uses its own capacity is therefore a large part of deciding which work belongs there, and Exhibit 5 shows how quickly the answer flips as utilization falls.
Dell, HPE, Cisco, Lenovo, and Supermicro give enterprises routes to AMD-based servers through established supplier relationships. Their offerings differ, including which systems support Instinct acceleration, so customers choose a supported configuration rather than a generic AMD platform. These partners also help size installations and work through the power, cooling, network, and support requirements of an existing data center, and the same equipment may run on premises or in a private deployment at a colocation facility. [6, 7, 14]
We also see a reason to plan for expansion without buying everything at once. Preparing a site for a later upgrade can avoid disruption, but unused capacity still costs money, so the customer needs a plausible view of the workloads and equipment it will add. A modular installation lets it expand as demand develops instead of committing immediately to the largest system it might eventually need.
Measure the cost of correctly completed work
We would start by dividing the full cost of the service by the number of tasks completed to the required standard and within the required time (Exhibit 6). That cost includes failed attempts, retries, and human review, since the customer pays for those along with the successful runs. Counting only accepted results keeps additional attempts from making a system look more productive than it is.
Power is a real limit on how much service an operator can deliver. Tokens per second per watt compares model serving under matched conditions. Energy per correctly completed task also captures the work around the model. More useful output within the same power cap can let an operator serve additional demand before a new facility is available. An agents-per-watt measure helps only when it describes active work and completed output, since a count of waiting environments does not establish productive capacity.
The customer should track both typical response time and slow requests. The median, or p50, describes the middle request, and the p99 is the response time within which 99 percent of requests finish. A system can look good at the median and still leave too many users waiting when queues build (Exhibit 7). Memory capacity, cache behavior, and scheduling influence those delays, so they belong in the comparison alongside peak throughput. More memory helps when it relieves a real working-set limit, but it is not evidence on its own that every AMD configuration delivers better tail latency.
CPU performance reaches those measurements in two different ways. Faster host processing can raise GPU output when request handling or input preparation is the bottleneck, and vLLM documents scaling its API servers for that situation. Faster tools can instead shorten an individual agent's task while the GPU fleet stays busy serving other users. Both improvements have value, but only the first addresses a direct limit on feeding the accelerator. [13, 15]
A full-system comparison combines these effects. The customer needs to know how many tasks finish correctly within the required time, what resources they consume, and what it costs to keep the service running. Measured that way, CPU execution, GPU serving, memory, and networking are all judged against the same result, and no one has to assume that improving a single component improves every workload.
Our view
We expect common interfaces and better serving software to keep widening the compute choices available to enterprises. Some of that choice already exists in hosted model services, where the application does not need to know the processor behind the endpoint, and customers operating their own infrastructure have another route through supported systems and familiar software platforms. Our compute-fungibility thesis depends on those choices becoming easier to use. It does not require every processor to become interchangeable.
AMD has several ways to participate: EPYC in the work around inference, Instinct in model serving, and networking and software that help customers use the capacity well. Helios gives partners an integrated design for large model workloads, and individual components reach the rest of the deployment. What we find most interesting about that position is that the supplier running the model does not automatically win every part of the work the model creates.
The opportunity grows as enterprises put agents into daily work. Our view depends on that activity justifying new capacity after efficiency gains and existing headroom are taken into account, and that is the part of our view that is hardest to model. Systems that maintain quality and response time while lowering the cost of completed work give customers a reason to expand, and AMD's portfolio and partners position it to compete for those purchases.
Sources
The compute-fungibility and CPU-execution discussion reflects Creative Strategies research and analysis. New and revised sources accessed September 29, 2026; retained sources are dated individually.
[1] AMD, "AAI 2026: AMD Delivers Full-Stack Compute for the Agentic AI Era," July 23, 2026. https://ir.amd.com/news-events/press-releases/detail/1294/aai-2026-amd-delivers-full-stack-compute-for-the-agentic-ai-era
[2] AMD Enterprise AI, AIMs Overview, version 2.2.2. https://enterprise-ai.docs.amd.com/en/v2.2.2/aims/overview.html
[3] AMD Enterprise AI, Reference Stacks. Supported configurations and deployment requirements apply. https://enterprise-ai.docs.amd.com/en/latest/reference-stacks.html
[4] Meta Engineering, "Scaling LLM Inference," October 17, 2025. https://engineering.fb.com/2025/10/17/ai-research/scaling-llm-inference-innovations-tensor-parallelism-context-parallelism-expert-parallelism/
[5] AMD ROCm 7.2.4 documentation, vLLM V1 performance optimization. Version-specific serving guidance, not a universal performance comparison. https://rocm.docs.amd.com/en/docs-7.2.4/how-to/rocm-for-ai/inference-optimization/vllm-optimization.html
[6] Dell, "Dell and AMD Are Expanding What's Possible for On-Premises AI," May 7, 2026. https://www.dell.com/en-us/blog/dell-and-amd-are-expanding-what-s-possible-for-on-premises-ai/
[7] HPE, HPE and AMD compute solutions portfolio. https://www.hpe.com/us/en/solutions/amd.%20html.html
[8] OpenRouter, Provider Routing. Endpoint selection is distinct from hardware qualification. https://openrouter.ai/docs/guides/routing/provider-selection
[9] AMD, Helios overview and open software ecosystem. https://www.amd.com/en/products/rackscale-solutions/helios.html
[10] NVIDIA, NIM for LLMs 1.12 API reference. Common API support does not establish identical model or feature behavior. https://docs.nvidia.com/nim/large-language-models/1.12.0/api-reference.html
[11] AMD, "The Importance of Open Ecosystems for Agentic AI Deployment," July 30, 2026. https://www.amd.com/en/blogs/2026/the-importance-of-open-ecosystems-for-agentic-ai-development.html
[12] LiteLLM Router, and vLLM Production Stack Semantic Router integration. The latter is labeled preview in the documentation reviewed. https://docs.litellm.ai/docs/routing; https://docs.vllm.ai/projects/production-stack/en/latest/use_cases/semantic-router-integration.html
[13] Creative Strategies, Agentic CPU Guidebook V5, September 28, 2026. CPU pools, execution time, memory, and provisioning analysis.
[14] AMD, EPYC OEM portfolio information. https://www.amd.com/content/dam/amd/en/documents/epyc-business-docs/infographics/cpu-you-choose-can-make-or-break-your-investment.pdf
[15] vLLM 0.25.1, Optimization and Tuning. https://docs.vllm.ai/en/v0.25.1/configuration/optimization/
Exhibits 4, 5, and 7 are Creative Strategies illustrations built on stated or hypothetical inputs to show a mechanism. They are not benchmarks, measured results, or forecasts.
© 2026 Creative Strategies. Forecasts are Creative Strategies outlooks, not AMD guidance. Product capabilities are described from public AMD documentation and are not claims of universal performance or measured customer savings.