An Exabyte a Day: Throughput-Oriented, Large Scale, Managed Data Transfers with Effingo
SIGCOMM 2024(2024)
摘要
WAN bandwidth is never too broad --- and the speed of light stubbornly constant. These two fundamental constraints force globally-distributed systems to carefully replicate data close to where they are processed or served. A large organization owning such systems adds dimensions of complexity with ever-changing network topologies, strict requirements on failure domains, multiple competing transfers, and layers of software and hardware with multiple kinds of quotas. We present Effingo, a throughput-oriented, massively-parallel data copy service we built at Google. For its users, Effingo delivers high-throughput transfers with an scp-like interface. For Google, Effingo optimizes the network cost with a small footprint on datacenters. We experimentally show how Effingo achieves fairness and efficiency through copy tree optimization and dynamic adaptation to changing network conditions. On a typical day, Effingo transfers over an exabyte of data between dozens of clusters spread across continents and serves more than 10,000 users.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要