Agentic AI workflows expose CPU-GPU bottlenecks in current enterprise server infrastructure, Microsoft and UT researchers find

Researchers from Microsoft Azure and the University of Texas at Austin found that conventional GPU-centric servers struggle with agentic AI. Because agentic workflows repeatedly cross the CPU-GPU boundary to coordinate tools and orchestration, they create severe processing bottlenecks. To address this, researchers proposed Agora, an architecture that dynamically reallocates resources, boosting CPU utilization by 30% and generation throughput by 82% in tests.

CPU-GPU coordination bottlenecks

• ▪Agentic AI workflows repeatedly cross the CPU-GPU boundary, placing the host CPU on the critical path because orchestration software runs on the host while inference runs on GPUs.
• ▪In agentic AI workflows, either the CPU or GPU can become a transient bottleneck, leaving both processors underutilized on average.
• ▪Researchers from Microsoft Azure and the University of Texas at Austin found that conventional GPU-centric servers struggle to efficiently execute multi-step agentic AI workflows.
• ▪Conventional server architectures strand CPU and GPU resources during agentic workloads because static resource provisioning is poorly matched to fragmented execution patterns.

Fragmented agentic execution patterns

• ▪Production telemetry and experiments show that AI agents spend significant time coordinating models, tools, and orchestration software rather than executing standalone LLM requests.
• ▪A controlled study of the CORAL framework showed a single agentic workload expanded into 580 LLM calls and 552 tool invocations, repeatedly ping-ponging between processors.
• ▪According to analyst Sanchit Vir Gogia, tool execution time matched or exceeded model inference time in more than 27 percent of agentic AI requests.
• ▪During agentic AI execution, host CPU utilization remains low before spiking during tool execution, while GPU utilization varies widely, leaving some accelerators idle.

Agora dynamic resource allocation

• ▪Microsoft and University of Texas researchers proposed Agora, a server architecture that dynamically reallocates CPU and GPU resources based on agentic workload behavior.
• ▪The proposed Agora architecture pools CPU cores by role, applies affinity-aware scheduling, and oversubscribes GPU memory to improve resource utilization.
• ▪In evaluations, the Agora architecture increased host CPU utilization by approximately 30 percent and increased generation throughput by 82 percent.
• ▪The Agora architecture reduced agentic tail latency by 2.5 times and freed roughly one-third of GPUs through workload consolidation.

Workflow-based infrastructure procurement

• ▪Greyhound Research analyst Sanchit Vir Gogia stated that enterprises must evaluate agentic AI as distributed applications with embedded inference rather than larger chatbots.
• ▪Analyst Sanchit Vir Gogia advised organizations to procure heterogeneous server workflows rather than sizing infrastructure based solely on individual processors or model inference.

1 source