SNAG: SDN-Managed Network Architecture for GridFTP Transfers Using Application-Awareness

Increasingly, academic campus networks support large-scale data transfer workflows for data-intensive science. These data transfers rely on high-performance, scalable, and reliable protocols for moving large amounts of data over a high-bandwidth, high-latency network. GridFTP is a widely used protocol for wide area network (WAN) data movement. However, as the GridFTP protocol does not share connection information with the network-layer, network operators have reduced flexibility, particularly in identifying/managing flows across the network. We address this problem by deploying a production “ application-aware ” software defined network (SDN) for managing GridFTP transfers for data-intensive science workflows. We first propose a novel application-aware architecture called SNAG (SDN-managed Network Architecture for GridFTP transfers). SNAG combines application-layer and network-layer collaboration (termed “application-awareness”) with SDN-enabled network management to classify, monitor and to manage network resources actively. Until now, our SNAG deployment has successfully classified over 1.5 Billion GridFTP connections at the Holland Computing Center (HCC), University of Nebraska-Lincoln (UNL). Next, we develop an application-aware SDN system to provide differentiated network services for distributed computing workflows. At HCC, we also demonstrate how our system ensures the quality of service (QoS) for high-throughput workflows such as Compact Muon Solenoid (CMS) and Laser Interferometer Gravitational-Wave Observatory (LIGO). Further, we also demonstrate how application-aware SDN can be exploited to create policy-driven approaches to achieve accurate resource accounting for each workflow. We present strategies for implementing differentiated network services and discuss their capacity improvement benefits. Lastly, we provide some guidelines and recommendations for developing application-aware SDN architectures for general-purpose applications.