The Physical Limits of Single-Site AI Training
As deep learning models scale toward trillions of parameters, AI infrastructure teams face a hard physical limit: single-site data center capacity. Power availability, cooling limits, and physical real estate frequently prevent operators from expanding GPU clusters within a single facility. To scale further, organizations must transition from traditional "scale-up" or "scale-out" designs within one building to a "scale-across" architecture that spans multiple physical data centers.
WhiteFiber, DriveNets, and WEKA have now validated a long-distance design that challenges that assumption. Their system connects GPU resources across an 83-kilometer span with 111.2 Tbps of aggregate bandwidth and sub-millisecond latency. The result is a practical reference architecture for expanding AI capacity across existing sites instead of waiting for one larger facility.
The Mechanics of Scale-Across AI Clustering
The primary barrier to multi-site GPU clustering is the synchronization phase of distributed training. During collective communication operations (such as All-Reduce), GPUs must exchange gradient updates. If the network link between data centers introduces significant latency or packet drops, GPUs sit idle, leading to low Model Flops Utilization (MFU).
The WhiteFiber architecture overcomes this by treating geographically separated data centers as a single logical fabric. This is achieved through three integrated layers:
- Optical Transport Layer (WhiteFiber): High-capacity, low-latency dark fiber links operating with advanced optical transceivers to sustain 111.2 Tbps of throughput over 83 km without intermediate regeneration.
- Disaggregated Network Fabric (DriveNets): A distributed disaggregated chassis (DDC) routing system that scales packet forwarding independently of control planes, optimizing traffic routing across the long-haul link.
- Distributed Data Platform (WEKA): A software-defined storage layer that presents a global namespace across both sites, ensuring low-latency data access and eliminating localized storage bottlenecks.
Managing Latency and Congestion Over 83 Kilometers
At 83 kilometers, the physical propagation delay of light in silica fiber introduces approximately 0.41 milliseconds of one-way latency (assuming ~5 microseconds per kilometer). Minimizing additional protocol and queuing overhead is critical to keeping the round-trip time (RTT) under the sub-millisecond threshold required for RoCEv2 (RDMA over Converged Ethernet).
At this distance, the round-trip propagation delay is about 0.82 ms, leaving roughly 41 MB of data in flight on every 400 Gbps port. That is enough to overwhelm shallow shared buffers during synchronized RoCEv2 bursts. The DriveNets fabric addresses the risk with deep-buffer switching, Priority Flow Control, ECN-based congestion signaling, and dynamic load balancing across optical paths. Those controls matter because a small loss rate can trigger retransmissions across thousands of GPUs and turn a fast WAN into an expensive training bottleneck.
| Parameter | Single-Site Baseline | WhiteFiber Scale-Across Architecture |
|---|---|---|
| Physical Distance | < 100 meters | Up to 83 kilometers |
| Interconnect Bandwidth | Local PCIe/NVLink speeds | 111.2 Tbps (WAN aggregate) |
| One-Way Fiber Latency | Nanoseconds | ~0.41 milliseconds (propagation delay) |
| Network Protocol | InfiniBand or RoCEv2 | RoCEv2 with optimized WAN buffers |
| Storage Namespace | Localized SAN/NAS | Unified WEKA Global Namespace |
Storage Synchronization and Data Consistency
A distributed GPU cluster is only as fast as its storage path. WEKA's global namespace lets either site read the same training data, while local caching keeps repeated reads close to GPUs. Operators must still test metadata latency, failure recovery, and consistency during link degradation.
Practical Takeaway
The real result is not headline bandwidth alone; it is the combination of optical capacity, deterministic congestion control, and a shared data plane. Before treating remote GPUs as one pool, teams should benchmark collective latency, buffer occupancy, storage failover, and cost per completed training run.
For organizations constrained by power, cooling, or real estate at one campus, this design offers a credible scale-out path. It coordinates separate facilities while keeping latency, resilience, and operational discipline measurable.
Sources
- WhiteFiber Announces Cross-Data-Center AI Infrastructure Innovation Delivering 111.2 Tbps Bandwidth with Sub-Millisecond Latency Across 83km — Web source, July 9, 2026
- DriveNets Announces Industry's First Commercial Deployment of a Long-Distance, Scale-Across AI Supercluster - PR Newswire — Web source, July 9, 2026