A Cross-Layer Congestion Control Mechanism for Shuffle-Intensive Analytics with Deadline-Aware Rate Adaptation

Abstract

Shuffle-intensive analytics frameworks commonly transform computation into alternating phases of local processing and all-to-all data exchange. In modern clusters, the shuffle phase frequently dominates both completion time and tail variability because it induces bursty many-to-many transfers, transient incast at reducers, and queue buildup along shared fabric bottlenecks. At the same time, interactive and production analytics are often executed under job- or stage-level deadlines, where the relevant objective is not only throughput but also timely delivery of intermediate partitions so that downstream tasks can begin without prolonged blocking. This text describes a cross-layer congestion control mechanism tailored to shuffle traffic with explicit deadline-aware rate adaptation. The mechanism couples application-layer knowledge of shuffle structure and remaining bytes with transport-layer pacing and explicit congestion feedback, allowing rates to be shaped by urgency while still respecting network stability. A coflow-oriented abstraction is used to associate multiple concurrent TCP-like flows with a single shuffle stage, exposing per-reducer and per-stage completion requirements. A lightweight price signal derived from ECN-marked congestion and receiver-side service pressure steers host rate controllers toward allocations that reduce queue occupancy and limit incast collapse. Deadline awareness is implemented by an urgency-weighted control law that increases rate when slack diminishes, but caps the response when congestion prices rise, yielding bounded queues and predictable fabric utilization. The overall design emphasizes deployability in commodity stacks through minimal switch features, kernel-level pacing hooks, and analytics runtime integration.