Simplifying The Deployment of Agentic AI at Scale

Author: Tim Zhao, Accton/Edgecore Collection: 2025 OCP Global Summit - Expo Hall Sessions

Summary

Accton/Edgecore presents their in-house experience deploying agentic AI at enterprise scale, addressing the operational challenges that make enterprise AI adoption difficult despite exciting technology breakthroughs. The presentation identifies five critical challenges: on-premises inference requirements driven by data jurisdiction, multi-tenancy requirements, coexistence with brownfield applications, lack of full-stack engineering expertise within enterprises, and long-term support and maintenance costs. These challenges are validated by customer feedback from Fortune 500 retailers (lacking full-stack capabilities), IT companies (different data security models for AI vs. traditional), and automotive companies (air-gapped, no internet access environments).

The proposed architecture follows an operations-first design philosophy with three layers: backend network (NVLink, PCIe, UALink), frontend Ethernet network with various NOS options, and end-to-end management/controller for integrated day-0 through day-2 operations. The key design principle is composable compute and storage -- software-defined GPUs, storage, and memory that can be dynamically allocated across departments to handle workload spikes from training, fine-tuning, and inference. The deployment methodology proceeds through day-0 (capacity and multi-tenancy planning), day-1 (deterministic deployment with rack power-on to model-ready in 60 minutes using automated playbooks with small/medium/large templates), and day-2 (steady-state operations with combined compute and network monitoring). A baseline validation approach using a 10-billion parameter Llama model ensures consistent post-deployment verification.