Ahmad Alhilal
Ze Wu
ABSTRACT
VR cloud gaming enables users to play high-end VR games on lightweight devices by offloading rendering tasks to cloud servers. Despite video compression, high-definition video streaming requires substantial data transfer rates. Foveated rendering (FR) and video encoding (FVE) leverage the non-uniform perception of the human visual system to reduce computing and bandwidth demand. They enhance visual quality in central gaze regions and reduce it in the periphery. However, bandwidth variation may hinder the provision of smooth VR gaming experiences. We present FovOptix, a system that combines FR with adaptive FVE to deliver video stream at a lower yet adaptive bitrate while not compromising the perceived video quality. FovOptix is based on a game-agnostic open-source to ensure reproducibility and compatibility with various games. We evaluate FovOptix against benchmarks using 5G mobile network traces. FovOptix achieves a latency reduction of 3% compared to the Google standard and a significant +100% reduction compared to other solutions. Additionally, it enhances the visual quality within the player's region of interest. Consequently, FovOptix attains the highest playability and gaming scores while minimizing the severity of motion sickness. FovOptix thus offers smooth and accessible VR cloud gaming for a wider range of players.
- 2018. SteamVR. https://store.steampowered.com/app/250820/SteamVR/Google Scholar
- 2023. Air Light VR (ALVR). https://github.com/alvr-org/ALVRGoogle Scholar
- 2023. NVENC Video Encoder API Programming Guide. https://docs.nvidia.com/video-technologies/video-codec-sdk/12.1/nvenc-video-encoder-api-prog-guide/index.htmlGoogle Scholar
- 2023. Real-time Communication for the Web (WebRTC). https://webrtc.org/Google Scholar
- Mehmet N. Akcay. 2021. Improving Server and Client-Side Algorithms for Adaptive Streaming of Non-Immersive and Immersive Media. In Proceedings of the 12th ACM Multimedia Systems Conference (Istanbul, Turkey) (MMSys '21). Association for Computing Machinery, New York, NY, USA, 383--387. https://doi.org/10.1145/3458305.3478461Google ScholarDigital Library
- Rachel Albert, Anjul Patney, David Luebke, and Joohwan Kim. 2017. Latency Requirements for Foveated Rendering in Virtual Reality. ACM Trans. Appl. Percept. 14, 4, Article 25 (sep 2017), 13 pages. https://doi.org/10.1145/3127589Google ScholarDigital Library
- Ahmad Alhilal, Tristan Braud, Bo Han, and Pan Hui. 2022. Nebula: Reliable Low-Latency Video Transmission for Mobile Cloud Gaming. In Proceedings of the ACM Web Conference 2022 (Virtual Event, Lyon, France) (WWW '22). Association for Computing Machinery, New York, NY, USA, 3407--3417. https://doi.org/10.1145/3485447.3512276Google ScholarDigital Library
- D.A. Atchison. 2023. Optics of the Human Eye. CRC Press. https://books.google.com.hk/books?id=5-WtEAAAQBAJGoogle Scholar
- Gaetano Carlucci, Luca De Cicco, Stefan Holmer, and Saverio Mascolo. 2016. Analysis and Design of the Google Congestion Control for Web Real-Time Communication (WebRTC). In Proceedings of the 7th International Conference on Multimedia Systems (Klagenfurt, Austria) (MMSys '16). Association for Computing Machinery, New York, NY, USA, Article 13, 12 pages. https://doi.org/10.1145/2910017.2910605Google ScholarDigital Library
- Matthias Dick, Oliver Wellnitz, and Lars Wolf. 2005. Analysis of Factors Affecting Players' Performance and Perception in Multiplayer Games. In Proceedings of 4th ACM SIGCOMM Workshop on Network and System Support for Games (Hawthorne, NY) (NetGames '05). Association for Computing Machinery, New York, NY, USA, 1--7. https://doi.org/10.1145/1103599.1103624Google ScholarDigital Library
- Florian Frieß, Matthias Braun, Valentin Bruder, Steffen Frey, Guido Reina, and Thomas Ertl. 2021. Foveated Encoding for Large High-Resolution Displays. IEEE Transactions on Visualization and Computer Graphics 27, 2 (2021), 1850--1859. https://doi.org/10.1109/TVCG.2020.3030445Google ScholarCross Ref
- Sandra G Hart. 2006. NASA-task load index (NASA-TLX); 20 years later. In Proceedings of the human factors and ergonomics society annual meeting, Vol. 50. Sage publications Sage CA: Los Angeles, CA, 904--908.Google ScholarCross Ref
- Jan Horský. 2022. Crowdsourcing VR headset data VR headset database. https://www.infinite.cz/projects/HMD-tester-virtual-reality-headset-database-utilityGoogle Scholar
- Sudeng Hu, Hanli Wang, and Sam Kwong. 2012. Adaptive Quantization-Parameter Clip Scheme for Smooth Quality in H.264/AVC. IEEE Transactions on Image Processing 21 (04 2012), 1911--1919. https://doi.org/10.1109/TIP.2011.2176347Google ScholarDigital Library
- Chun-Ying Huang, Cheng-Hsin Hsu, Yu-Chun Chang, and Kuan-Ta Chen. 2013. GamingAnywhere: An Open Cloud Gaming System. In Proceedings of the 4th ACM Multimedia Systems Conference (Oslo, Norway) (MMSys '13). Association for Computing Machinery, New York, NY, USA, 36--47. https://doi.org/10.1145/2483977.2483981Google ScholarDigital Library
- Bert Hubert. [n.d.]. Linux TC Man Page. https://linux.die.net/man/8/tcGoogle Scholar
- Gazi Karam Illahi, Thomas Van Gemert, Matti Siekkinen, Enrico Masala, Antti Oulasvirta, and Antti Ylä-Jääski. 2020. Cloud Gaming with Foveated Video Encoding. ACM Trans. Multimedia Comput. Commun. Appl. 16, 1, Article 7 (feb 2020), 24 pages. https://doi.org/10.1145/3369110Google ScholarDigital Library
- Gazi Karam Illahi, Matti Siekkinen, Teemu Kämäräinen, and Antti Ylä-Jääski. 2020. On the Interplay of Foveated Rendering and Video Encoding. In Proceedings of the 26th ACM Symposium on Virtual Reality Software and Technology (Virtual Event, Canada) (VRST '20). Association for Computing Machinery, New York, NY, USA, Article 66, 3 pages. https://doi.org/10.1145/3385956.3422126Google ScholarDigital Library
- Gazi Karam Illahi, Matti Siekkinen, Teemu Kämäräinen, and Antti Ylä-Jääski. 2021. Foveated Streaming of Real-Time Graphics. In Proceedings of the 12th ACM Multimedia Systems Conference (Istanbul, Turkey) (MMSys '21). Association for Computing Machinery, New York, NY, USA, 214--226. https://doi.org/10.1145/3458305.3463383Google ScholarDigital Library
- Yoshio Ishiguro and Jun Rekimoto. 2011. Peripheral vision annotation: noninterference information presentation method for mobile augmented reality. In Proceedings of the 2nd Augmented Human International Conference. 1--5.Google ScholarDigital Library
- Nuwan Janaka, Chloe Haigh, Hyeongcheol Kim, Shan Zhang, and Shengdong Zhao. 2022. Paracentral and Near-Peripheral Visualizations: Towards Attention-Maintaining Secondary Information Presentation on OHMDs during in-Person Social Interactions. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22). Association for Computing Machinery, New York, NY, USA, Article 551, 14 pages. https://doi.org/10.1145/3491102.3502127Google ScholarDigital Library
- Junchen Jiang, Vyas Sekar, and Hui Zhang. 2012. Improving Fairness, Efficiency, and Stability in HTTP-Based Adaptive Video Streaming with FESTIVE. In Proceedings of the 8th International Conference on Emerging Networking Experiments and Technologies (Nice, France) (CoNEXT '12). Association for Computing Machinery, New York, NY, USA, 97--108. https://doi.org/10.1145/2413176.2413189Google ScholarDigital Library
- Xing Liu, Bo Han, Feng Qian, and Matteo Varvello. 2019. LIME: Understanding Commercial 360° Live Video Streaming Services. In Proceedings of the 10th ACM Multimedia Systems Conference (Amherst, Massachusetts) (MMSys '19). Association for Computing Machinery, New York, NY, USA, 154--164. https://doi.org/10.1145/3304109.3306220Google ScholarDigital Library
- Vignesh V Menon, Christian Feldmann, Hadi Amirpour, Mohammad Ghanbari, and Christian Timmerer. 2022. VCA: Video Complexity Analyzer. In Proceedings of the 13th ACM Multimedia Systems Conference (Athlone, Ireland) (MMSys '22). Association for Computing Machinery, New York, NY, USA, 259--264. https://doi.org/10.1145/3524273.3532896Google ScholarDigital Library
- Lothar Pantel and Lars C. Wolf. 2002. On the Impact of Delay on Real-Time Multiplayer Games. In Proceedings of the 12th International Workshop on Network and Operating Systems Support for Digital Audio and Video (Miami, Florida, USA) (NOSSDAV '02). Association for Computing Machinery, New York, NY, USA, 23--29. https://doi.org/10.1145/507670.507674Google ScholarDigital Library
- Darijo Raca, Dylan Leahy, Cormac J. Sreenan, and Jason J. Quinlan. 2020. Beyond Throughput, the next Generation: A 5G Dataset with Channel and Context Metrics. In Proceedings of the 11th ACM Multimedia Systems Conference (Istanbul, Turkey) (MMSys '20). Association for Computing Machinery, New York, NY, USA, 303--308. https://doi.org/10.1145/3339825.3394938Google ScholarDigital Library
- Jihoon Ryoo, Kiwon Yun, Dimitris Samaras, Samir R. Das, and Gregory Zelinsky. 2016. Design and Evaluation of a Foveated Video Streaming Service for Commodity Client Devices. In Proceedings of the 7th International Conference on Multimedia Systems (Klagenfurt, Austria) (MMSys '16). Association for Computing Machinery, New York, NY, USA, Article 6, 11 pages. https://doi.org/10.1145/2910017.2910592Google ScholarDigital Library
- Jihoon Ryoo, Kiwon Yun, Dimitris Samaras, Samir R. Das, and Gregory Zelinsky. 2016. Design and Evaluation of a Foveated Video Streaming Service for Commodity Client Devices. In Proceedings of the 7th International Conference on Multimedia Systems (Klagenfurt, Austria) (MMSys '16). Association for Computing Machinery, New York, NY, USA, Article 6, 11 pages. https://doi.org/10.1145/2910017.2910592Google ScholarDigital Library
- Andrea Scupola, Alessandra Mastrocola, Paola Sasso, Romina Fasciani, Lucrezia Montrone, Benedetto Falsini, and Edoardo Abed. 2013. Assessment of Retinal Function Before and After Idiopathic Macular Hole Surgery. American Journal of Ophthalmology 156, 1 (2013), 132--139.e1. https://doi.org/10.1016/j.ajo.2013.02.007Google ScholarCross Ref
- Hans Strasburger, Ingo Rentschler, and Martin Jüttner. 2011. Peripheral vision and pattern recognition: A review. Journal of Vision 11, 5 (Dec. 2011), 13--13. https://doi.org/10.1167/1L5.13_eprint: https://arvojournals.org/arvo/content_public/journal/jov/933487/jov-11-5-13.pdf.Google ScholarCross Ref
- M. Tun, K.K. Loo, and J. Cosmas. 2008. Rate control algorithm based on quality factor optimization for Dirac video codec. Signal Processing: Image Communication 23, 9 (2008), 649--664. https://doi.org/10.1016/j.image.2008.07.003Google ScholarDigital Library
Index Terms
- FovOptix: Human Vision-Compatible Video Encoding and Adaptive Streaming in VR Cloud Gaming
Recommendations
Cloud Gaming with Foveated Video Encoding
Cloud gaming enables playing high-end games, originally designed for PC or game console setups, on low-end devices such as netbooks and smartphones, by offloading graphics rendering to GPU-powered cloud servers. However, transmitting the high-resolution ...
An HVS based adaptive quantization scheme for the compression of color images
In this paper a Human Visual System based adaptive quantization scheme is proposed. The proposed algorithm supports perceptually lossless as well as lossy compression. The algorithm uses a transform based compression approach using the wavelet transform,...
A new perceptual quality metric for video transrating for mobile devices
MCMC '10: Proceedings of the 2010 ACM multimedia workshop on Mobile cloud media computingIn video streaming over heterogeneous networks to mobile devices, severe bandwidth mismatch creates significant challenges in maintaining quality of experience for end users. The reduction of the video bit rate to address the problem of bandwidth ...
Comments