Bruce Spang
Nick McKeown
ABSTRACT
When developing a new networking algorithm, it is established practice to run a randomized experiment, or A/B test, to evaluate its performance. In an A/B test, traffic is randomly allocated between a treatment group, which uses the new algorithm, and a control group, which uses the existing algorithm. However, because networks are congested, both treatment and control traffic compete against each other for resources in a way that biases the outcome of these tests. This bias can have a surprisingly large effect; for example, in lab A/B tests with two widely used congestion control algorithms, the treatment appeared to deliver 150% higher throughput when used by a few flows, and 75% lower throughput when used by most flows---despite the fact that the two algorithms have identical throughput when used by all traffic.
Beyond the lab, we show that A/B tests can also be biased at scale. In an experiment run in cooperation with Netflix, estimates from A/B tests mistake the direction of change of some metrics, miss changes in other metrics, and overestimate the size of effects. We propose alternative experiment designs, previously used in online platforms, to more accurately evaluate new algorithms and allow experimenters to better understand the impact of congestion on their tests.
- Alberto Abadie, Joshua Angrist, and Guido Imbens. 2002. Instrumental Variables Estimates of the Effect of Subsidized Training on the Quantiles of Trainee Earnings. Econometrica 70, 1 (Jan. 2002), 27.Google Scholar
Cross Ref
- A. Aggarwal, S. Savage, and T. Anderson. 2000. Understanding the Performance of TCP Pacing. In Proceedings IEEE INFOCOM 2000. Conference on Computer Communications. Nineteenth Annual Joint Conference of the IEEE Computer and Communications Societies (Cat. No.00CH37064), Vol. 3. IEEE, Tel Aviv, Israel, 1157--1165. Google ScholarCross Ref
- Julia Alexander. 2020. Amazon and Apple Are Reducing Streaming Quality to Lessen Broadband Strain in Europe. (March 2020). https://www.theverge.com/2020/3/20/21188072/amazon-prime-video-reduce-stream-quality-broadband-netflix-youtube-coronavirusGoogle Scholar
- Peter M. Aronow and Cyrus Samii. 2017. Estimating average causal effects under general interference, with application to a social network experiment. Ann. Appl. Stat. 11, 4 (12 2017), 1912--1947. Google ScholarCross Ref
- Rukshani Athapathu, Ranysha Ware, Aditya Abraham Philip, Srinivasan Seshan, and Justine Sherry. 2020. Prudentia: Measuring Congestion Control Harm on the Internet. 2. http://www.justinesherry.com/papers/athapathu-n2women20.pdfGoogle Scholar
- Susan Athey, Dean Eckles, and Guido W. Imbens. 2018. Exact p-Values for Network Interference. J. Amer. Statist. Assoc. 113, 521 (2018), 230--240. arXiv:https://doi.org/10.1080/01621459.2016.1241178 Google ScholarCross Ref
- Pat Bajari, Brian Burdick, Guido Imbens, James McQueen, Thomas Richardson, and Ido Rosen. 2019. Multiple Randomization Designs for Interference. (2019). https://assets.amazon.science/c1/94/0d6431bf46f7978295d245dd6e06/double-randomized-online-experiments.pdfGoogle Scholar
- H. Balakrishnan, V. N. Padmanabhan, S. Seshan, M. Stemm, and R. H. Katz. 1998. TCP Behavior of a Busy Internet Server: Analysis and Improvements. In Proceedings. IEEE INFOCOM '98, the Conference on Computer Communications. Seventeenth Annual Joint Conference of the IEEE Computer and Communications Societies. Gateway to the 21st Century (Cat. No. 98, Vol. 1. 252--262 vol.1. Google ScholarCross Ref
- Guillaume Basse, Avi Feller, and Panagiotis Toulis. 2019. Randomization tests of causal effects under interference. Biometrika 106, 2 (02 2019), 487--494. arXiv:https://academic.oup.com/biomet/article-pdf/106/2/487/28575447/asy072.pdf Google ScholarCross Ref
- Guillaume W. Basse, Hossein Azari Soufiani, and Diane Lambert. 2016. Randomization and The Pernicious Effects of Limited Budgets on Auction Experiments. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016, Cadiz, Spain, May 9-11, 2016 (JMLR Workshop and Conference Proceedings), Arthur Gretton and Christian C. Robert (Eds.), Vol. 51. JMLR.org, 1412--1420. http://proceedings.mlr.press/v51/basse16b.htmlGoogle Scholar
- Neda Beheshti, Yashar Ganjali, Monia Ghobadi, Nick McKeown, and Geoff Salmon. 2008. Experimental Study of Router Buffer Sizing. In Proceedings of the 8th ACM SIGCOMM Conference on Internet Measurement (IMC '08). ACM, New York, NY, USA, 197--210. Google ScholarDigital Library
- Neda Beheshti, Petr Lapukhov, and Yashar Ganjali. 2019. Buffer Sizing Experiments at Facebook. In Proceedings of the 2019 Workshop on Buffer Sizing. ACM, Palo Alto CA USA, 1--6. Google ScholarDigital Library
- Thomas Blake and Dominic Coey. 2014. Why Marketplace Experimentation is Harder than It Seems: The Role of Test-Control Interference. In Proceedings of the Fifteenth ACM Conference on Economics and Computation (EC '14). Association for Computing Machinery, New York, NY, USA, 567--582. Google ScholarDigital Library
- Iavor Bojinov, David Simchi-Levi, and Jinglong Zhao. 2021. Design and Analysis of Switchback Experiments. arXiv:2009.00148 [stat] (Jan. 2021). arXiv:stat/2009.00148 http://arxiv.org/abs/2009.00148Google Scholar
- Bob Briscoe. 2007. Flow Rate Fairness: Dismantling a Religion. ACM SIGCOMM Computer Communication Review 37, 2 (March 2007), 63--74. Google ScholarDigital Library
- Yi Cao, Arpit Jain, Kriti Sharma, Aruna Balasubramanian, and Anshul Gandhi. 2019. When to Use and When Not to Use BBR: An Empirical Analysis and Evaluation Study. In Proceedings of the Internet Measurement Conference. ACM, Amsterdam Netherlands, 130--136. Google ScholarDigital Library
- Neal Cardwell, Yuchung Cheng, C. Stephen Gunn, Soheil Hassas Yeganeh, and Van Jacobson. 2017. BBR: Congestion-Based Congestion Control. Commun. ACM 60, 2 (Jan. 2017), 58--66. Google ScholarDigital Library
- Neal Cardwell, Yuchung Cheng, Soheil Hassas Yeganeh, Priyaranjan Jha, Yousuk Seung, Kevin Yang, Ian Swett, Victor Vasiliev, Bin Wu, Luke Hsiao, Matt Mathis, and Van Jacobson. 2019. BBRv2: A Model-Based Congestion Control Performance Optimizations. (Nov. 2019). https://www.ietf.org/proceedings/106/slides/slides-106-iccrg-update-on-bbrv2-00Google Scholar
- Erik Carlsson and Eirini Kakogianni. 2018. Smoother Streaming with BBR. (Aug. 2018). https://engineering.atspotify.com/2018/08/31/smoother-streaming-with-bbr/Google Scholar
- Nicholas Chamandy. 2016. Experimentation in a Ridesharing Marketplace. (Dec 2016). https://eng.lyft.com/experimentation-in-a-ridesharing-marketplace-f75a9c4fcf01Google Scholar
- Bruno Crépon, Esther Duflo, Marc Gurgand, Roland Rathelot, and Philippe Zamora. 2013. Do Labor Market Policies Have Displacement Effects? Evidence from a Clustered Randomized Experiment *. The Quarterly Journal of Economics 128, 2 (May 2013), 531--580. Google ScholarCross Ref
- Nikos Diamantopoulos, Jeffrey Wong, David Issa Mattos, Ilias Gerostathopoulos, Matthew Wardrop, Tobias Mao, and Colin McFarland. 2019. Engineering for a Science-Centric Experimentation Platform. arXiv:1910.03878 [cs] (Oct. 2019). arXiv:cs/1910.03878 http://arxiv.org/abs/1910.03878Google Scholar
- Mo Dong, Tong Meng, Doron Zarchy, Engin Arslan, Yossi Gilad, P Brighten Godfrey, and Michael Schapira. 2018. PCC Vivace: Online-Learning Congestion Control. In NSDI. 15.Google Scholar
- Nandita Dukkipati, Matt Mathis, Yuchung Cheng, and Monia Ghobadi. 2011. Proportional Rate Reduction for TCP. In Internet Measurement Conference. 15.Google Scholar
- Nandita Dukkipati, Tiziana Refice, Yuchung Cheng, Jerry Chu, Tom Herbert, Amit Agarwal, Arvind Jain, and Natalia Sutin. 2010. An Argument for Increasing TCP's Initial Congestion Window. ACM SIGCOMM Computer Communication Review 40, 3 (June 2010), 26--33. Google ScholarDigital Library
- Eric Dumazet. 2013. Pkt_sched: Fq: Fair Queue Packet Scheduler [LWN.Net]. (Aug. 2013). https://lwn.net/Articles/564825/Google Scholar
- Eric Dumazet. 2013. Tcp: TSO Packets Automatic Sizing [LWN.Net]. (Aug. 2013). https://lwn.net/Articles/564979/Google Scholar
- Dean Eckles, Brian Karrer, and Johan Ugander. 2016. Design and Analysis of Experiments in Networks: Reducing Bias from Interference. Journal of Causal Inference 5, 1 (Feb. 2016). Google ScholarCross Ref
- Tobias Flach, Nandita Dukkipati, Andreas Terzis, Barath Raghavan, Neal Cardwell, Yuchung Cheng, Ankur Jain, Shuai Hao, Ethan Katz-Bassett, and Ramesh Govindan. 2013. Reducing Web Latency: The Virtue of Gentle Aggression. In SIGCOMM. 12.Google Scholar
- Ken Florance. 2020. Reducing Netflix Traffic Where It's Needed While Maintaining the Member Experience. (March 2020). https://about.netflix.com/en/news/reducing-netflix-traffic-where-its-neededGoogle Scholar
- Andrew Gelman and Jennifer Hill. 2006. Causal Inference Using Regression on the Treatment Variable. In Data Analysis Using Regression and Multilevel/Hierarchical Models. Cambridge University Press, Cambridge, 167--198. Google ScholarCross Ref
- Peter Glynn, Ramesh Johari, and Mohammad Rasouli. 2020. Adaptive experimental design with temporal interference: A maximum likelihood approach. arXiv preprint arXiv:2006.05591 (2020).Google Scholar
- Hadas Gold. 2020. Netflix and YouTube Are Slowing down in Europe to Keep the Internet from Breaking. (March 2020). https://www.cnn.com/2020/03/19/tech/netflix-internet-overload-eu/index.htmlGoogle Scholar
- Nirmal Govind. 2018. A/B Testing and Beyond: Improving the Netflix Streaming Experience with Experimentation and Data.... (June 2018). https://netflixtechblog.com/a-b-testing-and-beyond-improving-the-netflix-streaming-experience-with-experimentation-and-data-5b0ae9295bdfGoogle Scholar
- Ilya Grigorik. 2013. HTTP: HTTP/1.X - High Performance Browser Networking (O'Reilly). (2013). https://hpbn.co/http1x/#using-multiple-tcp-connectionsGoogle Scholar
- Ilya Grigorik and Surma. 2019. Introduction to HTTP/2 | Web Fundamentals. (Sept. 2019). https://developers.google.com/web/fundamentals/performance/http2#request_and_response_multiplexingGoogle Scholar
- Huan Gui, Ya Xu, Anmol Bhasin, and Jiawei Han. 2015. Network A/B Testing: From Sampling to Estimation. In Proceedings of the 24th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, Florence Italy, 399--409. Google ScholarDigital Library
- M. E. Halloran and C. J. Struchiner. 1995. Causal Inference in Infectious Diseases. Epidemiology (Cambridge, Mass.) 6, 2 (March 1995), 142--151. Google ScholarCross Ref
- David Holtz, Ruben Lobel, Inessa Liskovich, and Sinan Aral. 2020. Reducing Interference Bias in Online Marketplace Pricing Experiments. arXiv:2004.12489 [econ, stat] (April 2020). arXiv:econ, stat/2004.12489 http://arxiv.org/abs/2004.12489Google Scholar
- David Holtz, Ruben Lobel, Inessa Liskovich, and Sinan Aral. 2020. Reducing Interference Bias in Online Marketplace Pricing Experiments. (2020). arXiv:stat.ME/2004.12489Google Scholar
- Guanglei Hong and Stephen W. Raudenbush. 2006. Evaluating Kindergarten Retention Policy. J. Amer. Statist. Assoc. 101, 475 (Sept. 2006), 901--910. Google ScholarCross Ref
- Te-Yuan Huang, Ramesh Johari, Nick McKeown, Matthew Trunnell, and Mark Watson. 2014. A Buffer-Based Approach to Rate Adaptation: Evidence from a Large Video Streaming Service. In Proceedings of the 2014 ACM Conference on SIGCOMM. ACM, Chicago Illinois USA, 187--198. Google ScholarDigital Library
- Per Hurtig, Habtegebreil Haile, Karl-Johan Grinnemo, Anna Brunstrom, Eneko Atxutegi, Fidel Liberal, and Ake Arvidsson. 2018. Impact of TCP BBR on CUBIC Traffic: A Mixed Workload Evaluation. In 2018 30th International Teletraffic Congress (ITC 30). IEEE, Vienna, 218--226. Google ScholarCross Ref
- Geoff Huston. 2018. TCP and BBR. (May 2018). https://ripe76.ripe.net/presentations/10-2018-05-15-bbr.pdfGoogle Scholar
- Guido W. Imbens and Donald B. Rubin. 2015. Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction. Cambridge University Press, USA.Google Scholar
- Alexey Ivanov. 2020. Evaluating BBRv2 on the Dropbox Edge Network. arXiv:2008.07699 [cs] (Aug. 2020). arXiv:cs/2008.07699 http://arxiv.org/abs/2008.07699Google Scholar
- Ramesh Johari, Hannah Li, Inessa Liskovich, and Gabriel Weintraub. 2020. Experimental design in two-sided platforms: An analysis of bias. arXiv preprint arXiv:2002.05670 (2020).Google Scholar
- Matt Joras and Yang Chi. 2020. How Facebook Is Bringing QUIC to Billions. (Oct. 2020). https://engineering.fb.com/2020/10/21/networking-traffic/how-facebook-is-bringing-quic-to-billions/Google Scholar
- Matt Joras and Yang Chi. 2020. How Facebook Is Bringing QUIC to Billions. (Oct. 2020). https://engineering.fb.com/2020/10/21/networking-traffic/how-facebook-is-bringing-quic-to-billions/Google Scholar
- Arash Molavi Kakhki, Samuel Jero, David Choffnes, Cristina Nita-Rotaru, and Alan Mislove. 2017. Taking a Long Look at QUIC: An Approach for Rigorous Evaluation of Rapidly Evolving Transport Protocols. In Proceedings of the 2017 Internet Measurement Conference. ACM, London United Kingdom, 290--303. Google ScholarDigital Library
- Brian Karrer, Liang Shi, Monica Bhole, Matt Goldman, Tyrone Palmer, Charlie Gelman, Mikael Konutgan, and Feng Sun. 2020. Network Experimentation at Scale. arXiv:2012.08591 [cs, stat] (Dec. 2020). arXiv:cs, stat/2012.08591 http://arxiv.org/abs/2012.08591Google Scholar
- David Kastelman and Raghav Ramesh. 2018. Switchback Tests and Randomized Experimentation Under Network Effects at DoorDash. (Feb. 2018). https://medium.com/@DoorDash/switchback-tests-and-randomized-experimentation-under-network-effects-at-doordash-f1d938ab7c2aGoogle Scholar
- Ron Kohavi, Alex Deng, Brian Frasca, Toby Walker, Ya Xu, and Nils Pohlmann. 2013. Online Controlled Experiments at Large Scale. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, Chicago Illinois USA, 1168--1176. Google ScholarDigital Library
- Ron Kohavi, Diane Tang, and Ya Xu. 2020. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing (first ed.). Cambridge University Press. Google ScholarCross Ref
- Gautam Kumar, Nandita Dukkipati, Keon Jang, Hassan M. G. Wassel, Xian Wu, Behnam Montazeri, Yaogong Wang, Kevin Springborn, Christopher Alfeld, Michael Ryan, David Wetherall, and Amin Vahdat. 2020. Swift: Delay Is Simple and Effective for Congestion Control in the Datacenter. In Proceedings of the Annual Conference of the ACM Special Interest Group on Data Communication on the Applications, Technologies, Architectures, and Protocols for Computer Communication. ACM, Virtual Event USA, 514--528. Google ScholarDigital Library
- Ike Kunze, Jan Ruth, and Oliver Hohlfeld. 2020. Congestion Control in the Wild---Investigating Content Provider Fairness. IEEE Transactions on Network and Service Management 17, 2 (June 2020), 1224--1238. Google ScholarDigital Library
- Raul Landa, Lorenzo Saino, Lennert Buytenhek, and João Taveira Araújo. 2021. Staying Alive: Connection Path Reselection at the Edge. In NSDI 2021. 20.Google Scholar
- Adam Langley, Alistair Riddoch, Alyssa Wilk, Antonio Vicente, Charles Krasic, Dan Zhang, Fan Yang, Fedor Kouranov, Ian Swett, Janardhan Iyengar, Jeff Bailey, Jeremy Dorfman, Jim Roskind, Joanna Kulik, Patrik Westin, Raman Tenneti, Robbie Shade, Ryan Hamilton, Victor Vasiliev, Wan-Teh Chang, and Zhongyi Shi. 2017. The QUIC Transport Protocol: Design and Internet-Scale Deployment. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication (SIGCOMM '17). Association for Computing Machinery, New York, NY, USA, 183--196. Google ScholarDigital Library
- Charles F. Manski. 2013. Identification of treatment response with social interactions. The Econometrics Journal 16, 1 (2013), S1--S23. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1368-423X.2012.00368.x Google Scholar
- Hongzi Mao, Shannon Chen, Drew Dimmery, Shaun Singh, Drew Blaisdell, Yuandong Tian, Mohammad Alizadeh, and Eytan Bakshy. 2020. Real-World Video Adaptation with Reinforcement Learning. arXiv:2008.12858 [cs] (Aug. 2020). arXiv:cs/2008.12858 http://arxiv.org/abs/2008.12858Google Scholar
- Radhika Mittal, Vinh The Lam, Nandita Dukkipati, Emily Blem, Hassan Wassel, Monia Ghobadi, Amin Vahdat, Yaogong Wang, David Wetherall, and David Zats. 2015. TIMELY: RTT-Based Congestion Control for the Datacenter. In Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication. ACM, London United Kingdom, 537--550. Google ScholarDigital Library
- Whitney K. Newey and Kenneth D. West. 1987. A Simple, Positive Semi-Definite, Heteroskedasticity and Autocorrelation Consistent Covariance Matrix. Econometrica 55, 3 (1987), 703--708. Google ScholarCross Ref
- Garg Nitin. 2019. COPA Congestion Control for Video Performance. (Nov. 2019). https://engineering.fb.com/2019/11/17/video-engineering/copa/Google Scholar
- Samuel D. Oman and Esther Seiden. 1988. Switch-Back Designs. Biometrika 75, 1 (March 1988), 81--89. Google ScholarCross Ref
- James Robins. 1986. A New Approach to Causal Inference in Mortality Studies with a Sustained Exposure Period---Application to Control of the Healthy Worker Survivor Effect. Mathematical Modelling 7, 9-12 (Jan. 1986), 1393--1512. Google ScholarCross Ref
- Donald B Rubin. 2005. Causal inference using potential outcomes: Design, modeling, decisions. J. Amer. Statist. Assoc. 100, 469 (2005), 322--331.Google ScholarCross Ref
- Ahmed Saeed, Nandita Dukkipati, Vytautas Valancius, Vinh The Lam, Carlo Contavalli, and Amin Vahdat. 2017. Carousel: Scalable Traffic Shaping at End Hosts. In The Conference of the ACM Special Interest Group. ACM Press, 404--417. Google ScholarDigital Library
- Martin Saveski, Jean Pouget-Abadie, Guillaume Saint-Jacques, Weitao Duan, Souvik Ghosh, Ya Xu, and Edoardo M. Airoldi. 2017. Detecting Network Effects: Randomizing Over Randomized Experiments. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, Halifax NS Canada, 1027--1035. Google ScholarDigital Library
- Robert Sayre. 2008. Change Max-Persistent-Connections-per-Server to 6. (March 2008). https://bugzilla.mozilla.org/show_bug.cgi?id=423377Google Scholar
- Nate Schloss and Ben Maurer. 2017. This Browser Tweak Saved 60% of Requests to Facebook. (Jan. 2017). https://engineering.fb.com/2017/01/26/web/this-browser-tweak-saved-60-of-requests-to-facebook/Google Scholar
- Dominik Scholz, Benedikt Jaeger, Lukas Schwaighofer, Daniel Raumer, Fabien Geyer, and Georg Carle. 2018. Towards a Deeper Understanding of TCP BBR Congestion Control. In 2018 IFIP Networking Conference (IFIP Networking) and Workshops. IEEE, Zurich, Switzerland, 1--9. Google ScholarCross Ref
- Anant Shah. 2019. BBR Evaluation at a Large CDN. (Nov. 2019). https://blog.apnic.net/2019/11/01/bbr-evaluation-at-a-large-cdn/Google Scholar
- Steve Souders. 2008. Roundup on Parallel Connections. (March 2008). https://www.stevesouders.com/blog/2008/03/20/roundup-on-parallel-connections/Google Scholar
- Bruce Spang, Brady Walsh, Te-Yuan Huang, Tom Rusnock, Joe Lawrence, and Nick McKeown. 2019. Buffer Sizing and Video QoE Measurements at Netflix. In Proceedings of the 2019 Workshop on Buffer Sizing. ACM, Palo Alto CA USA. Google ScholarDigital Library
- Jerzy Splawa-Neyman, Dorota M Dabrowska, and TP Speed. 1990. On the application of probability theory to agricultural experiments. Essay on principles. Section 9. Statist. Sci. (1990), 465--472.Google Scholar
- Diane Tang, Ashish Agarwal, Deirdre O'Brien, and Mike Meyer. 2010. Overlapping Experiment Infrastructure: More, Better, Faster Experimentation. In KDD'10. 10.Google Scholar
- Eric J. Tchetgen Tchetgen and Tyler J. VanderWeele. 2012. On causal inference in the presence of interference. Statistical Methods in Medical Research 21 (2012), 55 -- 75.Google ScholarCross Ref
- Martin Tingley. 2018. Streaming Video Experimentation at Netflix: Visualizing Practical and Statistical Significance. (Sept. 2018). https://netflixtechblog.com/streaming-video-experimentation-at-netflix-visualizing-practical-and-statistical-significance-7117420f4e9aGoogle Scholar
- Linus Torvalds. [n. d.]. Tcp_input.c - Linux (v5.11-Rc5). ([n. d.]). https://github.com/torvalds/linux/blob/2ab38c17aac10bf55ab3efde4c4db3893d8691d2/net/ipv4/tcp_input.c#L873Google Scholar
- Donald F Towsley. 2015. TCP, Congestion Control. (Nov. 2015). http://gaia.cs.umass.edu/cs653/slides/tcp.pdfGoogle Scholar
- Belma Turkovic, Fernando A. Kuipers, and Steve Uhlig. 2019. Fifty Shades of Congestion Control: A Performance and Interactions Evaluation. arXiv:1903.03852 [cs] (March 2019). arXiv:cs/1903.03852 http://arxiv.org/abs/1903.03852Google Scholar
- Belma Turkovic, Fernando A. Kuipers, and Steve Uhlig. 2019. Interactions between Congestion Control Algorithms. In 2019 Network Traffic Measurement and Analysis Conference (TMA). 161--168. Google ScholarCross Ref
- Johan Ugander, Brian Karrer, Lars Backstrom, and Jon Kleinberg. 2013. Graph Cluster Randomization: Network Exposure to Multiple Universes. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '13). Association for Computing Machinery, New York, NY, USA, 329--337. Google ScholarDigital Library
- Ranysha Ware, Matthew K. Mukerjee, Srinivasan Seshan, and Justine Sherry. 2019. Beyond Jain's Fairness Index: Setting the Bar For The Deployment of Congestion Control Algorithms. In Proceedings of the 18th ACM Workshop on Hot Topics in Networks. ACM, Princeton NJ USA, 17--24. Google ScholarDigital Library
- Ranysha Ware, Matthew K. Mukerjee, Srinivasan Seshan, and Justine Sherry. 2019. Modeling BBR's Interactions with Loss-Based Congestion Control. In Proceedings of the Internet Measurement Conference. ACM, Amsterdam Netherlands, 137--143. Google ScholarDigital Library
- David X. Wei, Pei Cao, and Steven H. Low. 2006. TCP Pacing Revisited. (2006). http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.92.2658&rep=rep1&type=pdfGoogle Scholar
- Francis Y Yan, Hudson Ayers, Chenzhi Zhu, Sadjad Fouladi, James Hong, Keyi Zhang, Philip Levis, and Keith Winstein. 2020. Learning in Situ: A Randomized Experiment in Video Streaming. In NSDI. Santa Clara, CA, USA, 16. https://www.usenix.org/system/files/nsdi20-paper-yan.pdfGoogle Scholar
Index Terms
- Unbiased experiments in congested networks
Recommendations
AQM performance in multiple congested link networks
ACM-SE 43: Proceedings of the 43rd annual Southeast regional conference - Volume 2TCP Reno, the most widely used implementation on the Internet, uses retransmission timeouts and the receipt of three duplicate ACKs to detect packet loss in the network. In case of multiple congested links, it is important to detect packet losses as ...
Connections with multiple congested gateways in packet-switched networks part 1: one-way traffic
In this paper we explore the bias in TCP/IP networks against connections with multiple congested gateways. We consider the interaction between the bias against connections with multiple congested gateways, the bias of the TCP window modification ...
Management of parallel UBR flows over TCP in congested ATM networks
The Linux transmission control protocol (TCP) implementation consists of TCP Reno, NewReno and selective acknowledgement options. This has improved the throughput of TCP/IP over the Internet. However, TCP over plain ATM networks significantly suffers ...
Comments