COVID-19 and the Internet: Lessons Learned

Abstract The COVID-19 pandemic has disrupted the ‘real’ world and substantially impacted the virtual world and thus the Internet ecosystem. It has caused a significant exogenous shock that offers a wealth of natural experiments and produced new data about broadband, clouds, and the Internet in times of crisis. In this chapter, we characterise and evaluate the evolving impact of the global COVID-19 crisis on traffic patterns and loads and the impact of those on Internet performance from multiple perspectives. While we place a particular focus on deriving insights into how we can better respond to crises and better plan for the post-COVID-19 ‘new normal’, we analyse the impact on and the responses by different actors of the Internet ecosystem across different jurisdictions. With a focus on the USA and Europe, we examine the responses of both public and private actors, with the latter including content and cloud providers, content delivery networks, and Internet service providers (ISPs). This chapter makes two contributions: first, we derive lessons learned for a future post-COVID-19 world to inform non-networking spheres and policy-making; second, the insights gained assist the networking community in better planning for the future. Keywords COVID-19 Internet traffic Resilience Broadband Internet Exchange Point Content Delivery Network Clouds Citation Stocker, V., Lehr, W. and Smaragdakis, G. (2023), "COVID-19 and the Internet: Lessons Learned", Whalley, J., Stocker, V. and Lehr, W. (Ed.) Beyond the Pandemic? Exploring the Impact of COVID-19 on Telecommunications and the Internet, Emerald Publishing Limited, Bingley, pp. 17-69. https://doi.org/10.1108/978-1-80262-049-820231002 Publisher: Emerald Publishing Limited Copyright © 2023 Volker Stocker, William Lehr and Georgios Smaragdakis License Published by Emerald Publishing Limited. This chapter is published under the Creative Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial and non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at http://creativecommons.org/licences/by/4.0/legalcode. 2.1. Introduction The virus SARS-CoV-2 and the associated disease COVID-19, which the WHO declared a pandemic on 11 March 2020, turned the world upside down, resulting in countries across the globe issuing various forms of stay-at-home social distancing rules and closing in-person economic activity in an effort to stem the spread of the disease (European Centre for Disease Prevention and Control, 2022; WHO, 2022). Where possible, virtual encounters replaced physical ones, and social, educational, and commercial activity increasingly moved online during the course of the lockdown and ongoing pandemic (at least for those activities that could shift online).1 This dramatic shift had profound effects on our social and economic lives. Some will result in long-lasting changes, while others may be temporary crisis responses. Some of the effects and responses were anticipated, some are surprises, and others are evolving in real-time. COVID-19 has disrupted the ‘real’ world and has substantial implications for the virtual world and thus the Internet ecosystem. It caused a significant exogenous shock that offers a wealth of natural experiments and produced new data about broadband, clouds, and the Internet in times of crisis and enables testing of established and proposed hypotheses about the resilience and adaptability of the ecosystem. These unparalleled research opportunities for observing the interaction effects between the real and virtual worlds provide novel possibilities to evaluate how well today’s communications networks, services, and applications have responded to the increased and changing traffic loads and assess the evolving responses by private actors such as ISPs and content and cloud providers as well as governments. The natural experiment(s) afforded will continue to be mined and analysed for network provisioning/management and policy insights in years to come. In this chapter, we highlight emerging insights and explore the interaction effects between the real and virtual worlds. Our focus is on the USA and Europe and deriving lessons and insights into how we can better plan for the future post-COVID-19 ‘new normal’. Recognising the research potential of the ongoing crisis, we began collecting trade-press, blog posts, academic research, sundry white papers, and related materials that were publicly available and related to the performance and management of Internet infrastructure and services and user and policy responses as those evolved in real-time starting in the first quarter of 2020 when the extent, duration, and impact of the pandemic were uncertain. Our collection methods were not systematic but were informed by our long engagement in multidisciplinary research related to Internet technology, industry, and policy developments. The materials we collected numbered over 3,000 entries. Our initial review of these materials, presented here, focuses on identifying how the COVID-19 experience helps confirm what was known pre-COVID-19, what lessons are new, and what questions remain to be explored further. The remainder of this chapter is structured as follows. In Section 2.2, we provide an overview of the effects of COVID-19 on Internet traffic and explore how well the Internet has coped with the new demands and where specific weaknesses were revealed. Section 2.3 highlights the responses by policy-makers and by industry. Section 2.4 discusses the impact of COVID-19 in light of those responses for the Internet ecosystem and the lessons we take from our evaluation. Section 2.5 briefly sums up and concludes. 2.2. The Effect of COVID-19 on Internet Traffic The pandemic and the measures to contain the spread of the virus have fundamentally changed social and commercial activity. Unsurprisingly, it has caused sudden and unexpected increases and shifting patterns in Internet traffic and substantially changed usage patterns (Feldmann et al., 2020, 2021; Koeze & Popper, 2020; Labovitz, 2020a; OECD, 2020a, 2020b). One of the most significant changes relates to the location of Internet access as many individuals had to rely on their residential broadband connections to maintain their social and economic activity, for example, working, educating, and consuming entertainment from home. Moreover, many citizens changed locations, leaving city centers and (temporarily) moving to more rural or remote areas. The change of access point has emphasised the important role of residential broadband access and in-home networks (e.g. local home WiFi networks). Where these networks are not well-provisioned, rely on outdated hardware and software, or are not configured correctly, they can present performance bottlenecks that limit access to online services and a good-quality user experience. Additionally, the shift to at-home use led to a geographical dispersion of the access points from ‘aggregation points’ such as enterprise networks or university campus networks. Moreover, reduced mobility implied that even when mobile devices were used, they were often connected via local home WiFi networks, thus relieving mobile network traffic (Comcast, 2020; Feldmann et al., 2020; Lutu et al., 2020; Schlosser et al., 2020; The Economist, 2020; see also Apple, 20212; Ritchie et al., 2022). Thus, the traffic that ISPs needed to carry shifted from originating at business locations to residential locations, with the attendant shift in the utilisation of the ‘first-hop’ access network facilities used to provision such activity. For example, the typical away-from-home access connection (e.g. office building, school, etc.) aggregates access traffic for many users (employees, students, etc.) before connecting it to wider-area networks off-site (whether those be the public Internet or private networks) via business-grade connections which are typically provisioned and tariffed differently from mass-market (residential) fixed or mobile broadband connections. In addition, the change of access location has often been accompanied by a change in the access environment – for example, a workplace (or school) network environment optimised and specifically secured was replaced by local access from home and remote access via virtual private networks (VPNs) (Feldmann et al., 2021; World Bank, 2020). Because bricks and mortar retailing (see Whalley & Curwen, 2023, this volume) and other places like cinemas had to close during lockdown measures, offline entertainment and commercial activities like retail shopping, restaurants, gyms, and other offline activities migrated to the virtual sphere. The result of these shifts was higher traffic demands by residential broadband access users and shifts in usage and traffic patterns (e.g. Baumgartner, 2020; Cloudflare, 2021; Feldmann et al., 2020, 2021; Filipovic & Cervall, 2020; OpenVault, 2020a, 2020b, 2020c, 2020d; The Economist, 2020). The changes noted above would not have been possible with the pre-2000 Internet where most users accessed the Internet over low-speed, intermittent dial-up modem connections. In such a world, the opportunity to shift economic activity online would have been much more severely constrained. The basic networking infrastructure for enabling connectivity, the devices, the applications, the software, and the digital services used by businesses and consumers were much less capable and ubiquitously in use than they were in the years immediately preceding the onset of the pandemic. Over the past decade, significant changes have occurred in the Internet ecosystem, with perhaps the most important change being the shift to generally available broadband Internet access services offering data rates measured in the 10s to 100s of Megabits per second or faster3 and the wide availability of end-user devices and supporting applications and software capable of real-time video-conferencing and other interactive, multimedia applications. These evolutionary ecosystem changes set the stage for a shift from face-to-face physical interactions to virtual interactions for those with the right equipment, skills, networking infrastructure, and jobs. E-commerce also flourished, with growing shares of global commercial activity having moved online. A key demand driver for much of this investment was the growth in demand for over-the-top (OTT) video entertainment, and concurrently, growing demand for everywhere accessibility that fuelled simultaneous growth in streaming media (video and music services like YouTube, Netflix, and Spotify launched in 2005, 2007, and 2011, respectively) as well as real mobile broadband (e.g. smartphones after 2007 iPhone release and 4G LTE after 2010). Accommodating these changes required significant investment and adjustments by network and service providers across the Internet ecosystem.4 In addition to the investment in more capable broadband last-mile infrastructure, the need to deliver the surge in video traffic propelled the rise of Content Delivery Networks (CDNs) that increasingly sought to deploy (highly) distributed serving infrastructures to cache content closer to end-users. Distributed cloud and serving infrastructures have brought networked computing resources closer to end-users. In addition, they reflect a cloudification process by which a growing share of traffic is delivered via cloud-based systems. On top of that, the rise and rapid growth of geographically distributed interconnection facilities expanded options for where networks can meet, directly interconnect, and exchange traffic (e.g. via so-called Internet Exchange Points (IXPs)). Consequently, these developments have contributed to significant changes in the topology of the Internet. The Internet has become flatter with fewer hops between communicating endpoints – for example, between end-users (human-to-human), smart devices (machine-to-machine), or an end-user and the server where the content is stored or data is processed (human-to-machine).5 Due to these pre-COVID-19 developments, the Internet was already well-positioned to handle the COVID-19 pandemic’s sudden and forced shift from physical to virtual economic and social activity in many advanced, broadband-capable markets as a consequence of having been investing heavily in prior years to address the double-digit annual growth rates in traffic that have characterised the Internet for the past decades. 2.2.1. Impact on Application Usage The first question to ask is how the demands for different applications have changed compared to pre-pandemic levels. In other words, how much strain has the shift towards more virtual activities put on the providers of those applications that have been used during the pandemic, especially those that have acted as virtual substitutes for previously physical activities. Exploring this strain is crucial. We have already mentioned that real-world changes have caused adaptations and changes in the virtual world which, in turn, translate into changing demands for application usage and also for network traffic. Let us consider how real-world changes in developed countries have manifested themselves in the virtual world. In doing so, we first take a look at changes in Internet usage by application category as reported in numerous reports and studies as well as blog and news articles. The data is rich since over the course of the pandemic, especially during the first wave and the concomitant restrictions to contain the spread of the virus, a wide range of actors like ISPs (e.g. Comcast, 2020, 2021; Verizon, 2020), vendors like Sandvine (2020) or Nokia (Labovitz, 2020a), advisory groups like BITAG (2021), and also academics (e.g. Arkko et al., 2021) have published data and insights into the changing usage patterns. We analysed these sources and collected some of the reported changes in Table 2.1. Table 2.1. (Examples of) Changes in Internet Usage by Application Category. Application Category What? Change (Time Period; Location/Network) Source Entertainment Online Gaming Downloads (Network Usage) +20–80% (16 May 2020 vs. 1 Mar. 2020; Comcast) Comcast (2020) Data Usage/Traffic +82% (21 May 2020 vs. Typical Pre-pandemic Day; Verizon) Verizon (2020) +71% (22 Apr. 2020 vs. Typical Pre-pandemic Day [Peaks]; Verizon) +115% (9 Apr. 2020 vs. Typical Pre-pandemic Day; Verizon) +75% (16 Mar. 2020 vs. 9 Mar. 2020; Verizon) Video Streaming Consumption (Network Usage; Streaming & Web Video) +20–40% (16 May 2020 vs. 1 Mar. 2020; Comcast) Comcast (2020) Data Usage/Traffic +36% (14 May 2020 vs. Typical Pre-pandemic Day; Verizon) Verizon (2020) +26% (22 Apr. 2020 vs. Typical Pre-pandemic Day [Peaks]; Verizon) +36% (9 Apr. 2020 vs. Typical Pre-pandemic Day; Verizon) +12% (16 Mar. 2020 vs. 9 Mar. 2020; Verizon) Traffic Share (Absolute) Overall Traffic Share of Video Streaming: 57.64% (Apr. 2020; Global) Sandvine (2020) Netflix: 11.42% of Global Internet Traffic (Apr. 2020; Global) YouTube: 15.94% of Global Internet Traffic (Apr. 2020; Global) Social Media Facebook/Instagram/WhatsApp Usage >+50% Total messaging (in many countries ‘hit hardest’; ca. 24 Mar. 2020 vs. Previous month) Schultz and Parikh (2020) Up to +70% more time spent on apps (ca. 24 Mar. 2020 vs. Pre-pandemic Levels; Italy) >+1,000% Group call time (within one month; Data: ca. 24 Mar. 2020; Italy) +100% Live views on Instagram and Facebook (within one week; Data: ca. 24 Mar. 2020; Italy) Remote Work/Learning VPN Network Usage/Traffic +30–40% (16 May 2020 vs. 1 Mar. 2020, Weekdays; Comcast) Comcast (2020) Data Usage/Traffic +72% (21 May 2020 vs. Typical Pre-pandemic Day; Verizon) Verizon (2020) +81% (14 May 2020 vs. Typical Pre-pandemic Day; Verizon) +49% (9 Apr. 2020 vs. Typical Pre-pandemic Day; Verizon) +34% (16 Mar. 2020 vs. 9 Mar. 2020; Verizon) Traffic >+200% (Mar. 2020 vs. Mar. 2019, During Workdays & Working Hours; Different Vantage Points)a Feldmann et al. (2020) Online Collaboration Tools Data Usage/Traffic (Aggregate over all Tools) +1,200% (14 May 2020 vs. Typical pre-pandemic day; Verizon) Verizon (2020) Network Usage/Traffic (Voice over IP & Video Conferencing) +210–285% (16 May 2020 vs. 1 Mar. 2020, Weekdays; Comcast) Comcast (2020) Traffic (Web Conferencing and Telephony) >+200% (Mar. 2020 vs. Mar. 2019, During Workdays & Working Hours; Different Vantage Points)a Feldmann et al. (2020) Other Telehealth Utilization 38 Times Higher Utilisation Compared to Pre-pandemic Levels (Feb. 2021; Global) Bestsennyy et al. (2021) 78 Times Higher Utilisation Compared to Pre-pandemic Levels (Apr. 2020; Global) E-Commerce COVID-19 related additional growth in revenues +22% (2020–2021; Global [Forecast]) Statista (2021) +19% (2019–2020; Global) Change in the share of online retail sales of total retail sales (per country) +47.5% (2019–2020) versus +6.0% (2018–2019) in the UK Own calculation based on UNCTAD (2021) +27.3% (2019–2020) versus +11.1% (2018–2019) in the USA Change in gross merchandise value (GMV) per e-commerce company +38.0% (2019–2020) versus +21.0% (2018–2019) for Amazon UNCTAD (2021) +17.0% (2019–2020) versus –4.8% (2018–2019) for eBay +95.6% (2019–2020) versus +48.7% (2018–2019) for Shopify –37.1% (2019–2020) versus +29.3% (2018–2019) for Airbnb –63.3% (2019–2020) versus +4.0% (2018–2019) for Booking Holdings aThe authors utilise data from a set of vantage points: one large European ISP which operates a Tier-1 network, three IXPs (a major IXP in Central Europe; an IXP in Southern Europe; an IXP at the US East Coast), and one metropolitan educational network (REDImadrid). Table 2.1 gives a good idea of the nature and magnitude of the changes in the usage of different applications during the crisis. Whereas the applications and the usage changes presented are inherently selective, reflecting observations from different vantage points in the USA and Europe, they are broadly representative. A glance at the table yields three important insights. First, the demands for different applications have indeed changed through the pandemic, often dramatically. However, the changes varied strongly across applications and across providers. As Table 2.1 shows for different application categories, the demands for VPN services, social media, Telehealth, online gaming, video streaming, and online collaboration tools increased sharply, and even skyrocketed in some cases (see also Arkko et al., 2021; Feldmann et al., 2020, 2021; Jennings & Kozanian, 2020).6 In this context, it needs to be noted that applications with stringent latency requirements such as online gaming or video conferencing (as used for online collaboration in remote work or learning contexts or in Telehealth) have experienced dramatic growth rates. Video conferencing relies on bidirectional, real-time communications. Thus, these applications emphasise the role of upstream data rates and high and stable quality of service levels (especially regarding latency and jitter which are important to enable real-time interactivity of the sort required for video conferencing, gaming, and business ‘groupware’ applications). Specific services (such as corporate websites or databases used by employees) or VPNs that relied on centralised server architectures rather than on hosted-cloud solutions experienced significant congestion, especially on the up-links connecting end-users to centralised servers. In contrast, applications or VPNs that were provisioned using cloud services were better able to manage the demand shocks. Hosted-cloud solutions performed better since they were able to distribute the load and provide easier scalability options. For example, business applications like Office360 and Zoom’s video conferencing which are native cloud applications were better able to scale quickly and resiliently to meet localised COVID-19 traffic surges (Feldmann et al., 2020, 2021; see also Sections 2.3 and 2.4). Second, when taking a look at application categories as aggregates, Table 2.1 shows that shifts in the pre-pandemic application mix have emerged. For example, with employees working from home and children home from school, applications like video conferencing and group-sharing applications like Google Docs and Slack and video streaming and gaming applications saw significant jumps in usage, resulting in these accounting for a larger share of the application mix (see also Sandvine, 2020, p. 6). Although this is rather unsurprising, it has profound implications for the traffic experienced by networks over time and at different locations. The shifts in application mix varied by geographic location and over time for multiple reasons, including demographic and employment differences (across residential communities), differences in the progress of the pandemic and responses to it, or seasonal effects. These differences complicate the challenge of traffic analyses and of assessing the reasons behind the variable performance, to the extent such variances were observed. They demonstrate the need for differentiated assessments and evaluations of (i) the impact of the pandemic and (ii) the providers’ and networks’ resilience and ability to absorb and adapt to the changing demands. Third, demand shocks have posed challenges for application providers, content and cloud providers, and networks and communications service providers. The example of Zoom has shown that the demand shifts experienced by single application providers significantly exceeded the changes in terms of application categories or aggregate Internet traffic. As a consequence, the need to rapidly scale capacities and business operations capacities and adapt resource management strategies to handle localised hotspots (associated with particular applications at particular locations and times) while maintaining high levels of customer experience varies strongly across applications and service providers, and those differed with respect to their capabilities to accommodate the (unexpected) shifts in demand based on multiple factors, including their level of pre-pandemic investment, network architecture, and traffic management practices (e.g. how hot or close to peak capacity they typically ran their networks). Similarly, although aggregate web traffic increased, certain websites experienced traffic increases that were orders of magnitude larger (e.g. Berthene, 2020; Burke, 2020; Hendry, 2020; Koeze & Popper, 2020). The latter included sites providing such content as COVID-response-related material, including unemployment subsidy applications and COVID-19 testing information. This insight emphasises the relevance to perform differentiated analyses and consider the context- and locality-specific nature of the challenges by different actors of the ecosystem. 2.2.2. Impact on Internet Traffic Changing end-user and edge provider (e.g. application and content provider) usage patterns translate into changes in network traffic. They also imply changing requirements regarding network capacity and performance (e.g. in terms of reliability and latency, jitter, and packet loss rates). As we described above, the shift towards more virtual activities changed the locations from where, the timing for when, the selection of applications, and the modalities (e.g. type of device) online applications and services were used.7 These shifts resulted in commensurate shifts in network traffic, imposing strains on the ISP networks and ancillary service providers that connect end-users and application/content service providers. Against the background of the sudden demand shifts caused unexpectedly by the pandemic, it is hardly surprising that in the early stages of the lockdown measures, concerns arose that the Internet might collapse from the need to rapidly adjust to shifting so much activity online in response to COVID-19-related mandates (e.g. Fleming, 2020; Watson, 2020). Although ISPs had grown accustomed to aggregate traffic growth on the order of 30% per year, in March 2020, many ISPs and other providers experienced such levels of growth over a few weeks (e.g. Feldmann et al., 2020). As collective experience has demonstrated, the Internet has not collapsed but instead coped rather well with the unexpected increases and shifts in traffic. Already by mid-2020, the Internet had weathered the storm of the first wave and had proven its critical role in enabling online activity to substitute for offline activities, and in so doing, contributed significantly to mitigating the substantial negative social and economic effects of the pandemic that otherwise would have occurred had the pandemic struck in a world with less-advanced Internet capabilities (e.g. Belson, 2020a; Heaven, 2020; Stocker & Whalley, 2021; Timberg, 2020). Digital infrastructures, in particular the Internet, provided a lifeline for many and contributed significantly to social and economic resilience during the crisis (e.g. Briglauer & Stocker, 2020;8 Cloudflare, 2021; Feldmann et al., 2020, 2021; Rexford, 2021). With the trend towards telecommuting and more flexible work/schooling options (with mixed onsite and remote work/education) becoming increasingly prevalent, especially as the pandemic continues to cause restrictions that require the adoption of such options, COVID-19 has provided a significant step-change boost in support for and efforts to improve the robustness and capabilities of our broadband networks. In 2022, the question now is where the future post-COVID new normal will be and how will employers and schools adjust when onsite and in-person operations become increasingly acceptable. A series of studies and reports have investigated the stability, resilience, and adaptability of the Internet during the pandemic. Whereas many of these reports and studies had been motivated by the initial impact of the first wave and lockdowns across different countries and the sudden changes these have caused, some also covered the effects of subsequent waves and lockdown measures. Appendix 2.1 provides an overview of some of these studies and reports. 2.2.2.1. Internet Traffic and Network Performance – A Tale of Aggregates, Peaks, and Troughs To understand the pandemic’s impact on Internet traffic and network performance, it is important to understand the extent to which peak traffic and network utilisations change. As mentioned above and as shown in several studies, overall Internet traffic increased by 25–30% within a few couple of weeks and thus by as much as it would normally increase within an entire year (FCC, 2020b; Feldmann et al., 2020; Leighton, 2020b).9 This level of aggregate increase in demand in a few short weeks represents a significant demand shock that would stress many industries, and so for those not familiar with how networks are provisioned, it is hardly surprising that some feared the increased traffic might result in serious disruptions and degradation in Internet performance (Fleming, 2020; Timberg, 2020). However, the Internet has lived with double-digit annual aggregate (and per-average-user) traffic growth for several decades (e.g. Cisco, 2020) so the challenge for well-provisioned ISPs was to accommodate a year’s worth of growth in a few weeks – difficult, but not infeasible. Similar levels of traffic growth as were experienced by access provider ISPs were experienced by transit providers, cloud and content providers, and at IXPs (e.g. BITAG, 2021; Davidson, 2020; OECD, 2020a). In provisioning networks, capacity is added in lumpy increments in advance of projected demand growth. The reference point for upgrade decisions is oriented at traffic peaks and peak utilisation levels since this indicates network congestion. To address the unexpected COVID-19-related traffic spikes and to maintain high customer experience levels, ISPs needed to pull forward their annual capital spending and provisioning work by a number of months to allow them to accommodate the growth in traffic peaks (e.g. Liu et al., 2021). Whereas changes in peak traffic and peak utilisation are critical in assessing the impact on Internet performance, it is worthwhile to note that (i) decisions on capacity upgrades are based on expected growth in traffic peaks since these critically determine the stability and performance of network operations during peak demands and thus customer experience; and (ii) that providing for excess capacity during the peak to accommodate normal fluctuations in traffic is a standard operating procedure. However, the amount of excess peak capacity that is provisioned must be balanced with economic considerations. That is, too much excess capacity for unexpected peaks results in over-provisioning, excessively low average utilisation, and equivalently, high average costs and is – at least in normal times when average yearly growth rates are rather predictable at about 25 to 30% – economically inefficient. The fact that aggregate traffic growth alone does not give insights into the (potential) impact of the pandemic on network performance can be illustrated by a simple example. Suppose all traffic growth would have been in off-peak hours, that is, in pre-COVID-19 traffic troughs when network utilisation was very low anyway. In such a case, large traffic growths may not even require additional capacity investment. So, the questions to ask are when did the major increase in traffic occur and how have traffic peaks changed? In fact, much of the traffic increase occurred during off-peak periods (i.e. filling in pre-COVID-19 troughs).10 Thus, the strain on ISP capacity was significantly easier to accommodate than if aggregate increases had occurred at the peaks and been accommodated with a