Jia-Wei Fang
ABSTRACT
Cloud Virtual Reality (VR) gaming offloads the computationally-intensive rendering tasks from resource-limited Head-Mounted Displays (HMDs) to cloud servers, which consume a staggering amount of bandwidth for high-quality gaming experiences. One way to cope with such high bandwidth demands is to capitalize on human vision systems by allocating a higher bitrate to the foveal region of HMD viewport, which is known as foveation in the literature. Although foveation was employed by remote VR gaming, existing open-source projects all adopt static foveation, in which the HMD gamer gaze position is assumed to be fixed at the viewport center. In this paper, we construct the very first cloud VR gaming system that supports dynamic foveation. That is, the real-time gaze positions of gamers are streamed from eye-trackers on HMDs to cloud servers, which in turn adjust the foveation parameters, such as foveal region size/location and peripheral region quality degradation, accordingly. Using our developed cloud VR gaming system, we design and carry out a user study using a game called Fruit Ninja VR 2 to find the foveation parameters in static and dynamic foveation for maximizing the gaming Quality of Experience (QoE) in Mean Opinion Score (MOS). With the chosen foveation parameters, we found that, compared to cloud VR gaming without foveation, static foveation leads to a MOS increase of 0.60 and a bitrate reduction of 8.71%. Furthermore, adopting dynamic foveation results in an additional 0.60 increase on MOS while saving 9.81% bitrate, compared to static foveation. Our findings demonstrate the potential of dynamic foveation in cloud VR gaming, which dictates both high visual quality and short response time. The optimization techniques developed in this and follow-up work could benefit other cloud-rendered applications that typically have less strict requirements than cloud VR gaming.
- 2018. Cloud VR network solution white paper. https://reurl.cc/ml3OqV.Google Scholar
- 2020. Cloud gaming market size. https://reurl.cc/b7RLRM.Google Scholar
- 2020. The official website of NVIDIA CloudXR. https://reurl.cc/10djxm.Google Scholar
- 2020. Virtual reality market size, share & trends analysis report by technology. https://reurl.cc/MRdadk.Google Scholar
- 2021. The github of ALXR. https://github.com/korejan/ALVR/releases.Google Scholar
- 2021. PICO Neo 3 Pro Eye. https://reurl.cc/Nqgpep.Google Scholar
- 2022. Meta Quest Pro. https://reurl.cc/6NMEoV.Google Scholar
- 2023. OpenXR SDK document. https://reurl.cc/qkk393.Google Scholar
- R. Albert, A. Patney, D. Luebke, and J. Kim. 2017. Latency requirements for foveated rendering in virtual reality. ACM Transactions on Applied Perception 14, 4 (2017), 1--13.Google ScholarDigital Library
- F. Frieß, M. Braun, V. Bruder, S. Frey, G. Reina, and T. Ertl. 2020. Foveated encoding for large high-resolution displays. IEEE Transactions on Visualization and Computer Graphics 27, 2 (2020), 1850--1859.Google ScholarCross Ref
- B. Guenter, M. Finch, S. Drucker, D. Tan, and J. Snyder. 2012. Foveated 3D graphics. ACM Transactions on Graphics 31, 6 (2012), 1--10.Google ScholarDigital Library
- M. Hegazy, K. Diab, M. Saeedi, B. Ivanovic, I. Amer, Y. Liu, G. Sines, and M. Hefeeda. 2019. Content-aware video encoding for cloud gaming. In Proc. of ACM Multimedia Systems Conference (MMSys'19). 60--73.Google Scholar
- L. Hsiao, B. Krajancich, P. Levis, G. Wetzstein, and K. Winstein. 2022. Towards retina-quality VR video streaming: 15ms could save you 80% of your bandwidth. ACM SIGCOMM Computer Communication Review 52, 1 (2022), 10--19.Google ScholarDigital Library
- C. Hsu, A. Chen, C. Hsu, C. Huang, C. Lei, and K. Chen. 2017. Is foveated rendering perceivable in virtual reality? Exploring the efficiency and consistency of quality assessment methods. In Proc. of ACM International Conference on Multimedia (MM'17). 55--63.Google Scholar
- C. Huang, C. Hsu, Y. Chang, and K. Chen. 2013. GamingAnywhere: An open cloud gaming system. In Proc. of ACM Multimedia Systems Conference (MMSys'13). 36--47.Google Scholar
- G. Illahi, T. Gemert, M. Siekkinen, E. Masala, A. Oulasvirta, and A. Ylä-Jääski. 2020. Cloud gaming with foveated video encoding. ACM Transactions on Multimedia Computing, Communications, and Applications 16, 1 (2020), 1--24.Google ScholarDigital Library
- G. Illahi, M. Siekkinen,T. Kämäräinen, and A. Ylä-Jääski. 2021. Foveated streaming of real-time graphics. In Proc. of ACM Multimedia Systems Conference (MMSys'21). 214--226.Google Scholar
- T. Installations and L. Line. 1999. Subjective video quality assessment methods for multimedia applications. Networks 910, 37 (1999), 5.Google Scholar
- Y. Jin, M. Chen, T. Goodall, A. Patney, and A. Bovik. 2021. Subjective and objective quality assessment of 2D and 3D foveated video compression in virtual reality. IEEE Transactions on Image Processing 30 (2021), 5905--5919.Google ScholarDigital Library
- P. Kortum and W. Geisler. 1996. Implementation of a foveated image coding system for image bandwidth reduction. In Proc. of Human Vision and Electronic Imaging, Vol. 2657. 350--360.Google ScholarCross Ref
- I. Mohammadi, M. Hashemi, and M. Ghanbari. 2015. An object-based framework for cloud gaming using player's visual attention. In Proc. of IEEE International Conference on Multimedia & Expo Workshops (ICMEW'15). 1--6.Google Scholar
- A. Patney, M. Salvi, J. Kim, A. Kaplanyan, C. Wyman, N. Benty, D. Luebke, and A. Lefohn. 2016. Towards foveated rendering for gaze-tracked virtual reality. ACM Transactions on Graphics 35, 6 (2016), 1--12.Google ScholarDigital Library
- M. Romero-Rondón, L. Sassatelli, F. Precioso, and R. Aparicio-Pardo. 2018. Foveated streaming of virtual reality videos. In Proc. of ACM Multimedia Systems Conference (MMSys'18). 494--497.Google Scholar
- J. Ryoo, K. Yun, D. Samaras, S. Das, and G. Zelinsky. 2016. Design and evaluation of a foveated video streaming service for commodity client devices. In Proc. of ACM Multimedia Systems Conference (MMSys'16). 1--11.Google Scholar
- Sctanf. 2021. The linear and non-linear function of foveated warping method in ALVR. https://reurl.cc/3O5N10.Google Scholar
- S. Shi and C. Hsu. 2015. A survey of interactive remote rendering systems. Comput. Surveys 47, 4 (2015), 1--29.Google ScholarDigital Library
- F. Tong. 2018. Foundations of vision. Stevens' Handbook of Experimental Psychology and Cognitive Neuroscience 2 (2018), 1--61.Google Scholar
- Valve. 2023. SteamVR. https://www.steamvr.com/en/.Google Scholar
- W. Zou, S. Feng, X. Mao, F. Yang, and Z. Ma. 2021. Enhancing quality of experience for cloud virtual reality gaming: an object-aware video encoding. In Proc. of IEEE International Conference on Multimedia & Expo Workshops (ICMEW'21). 1--6.Google Scholar
Index Terms
- Will Dynamic Foveation Boost Cloud VR Gaming Experience?
Recommendations
Modeling Gamer Quality-of-Experience Using a Real Cloud VR Gaming Testbed
MMVE '23: Proceedings of the 15th International Workshop on Immersive Mixed and Virtual Environment SystemsCloud Virtual Reality (VR) gaming offloads computationally-intensive VR games to resourceful data centers. Ensuring good Quality of Experience (QoE) in cloud VR gaming, however, is inherently challenging as VR gamers demand high visual quality, short ...
Player Experience in a VR and Non-VR Multiplayer Game
VRIC '18: Proceedings of the Virtual Reality International Conference - Laval VirtualSeveral studies compared the player experience of Virtual Reality (VR) games and non-VR games. However, there is a lack of research on the specific subject of VR and non-VR multiplayer games. Therefore, this work explores how implementing a multiplayer ...
The Effect of Body-Based Haptic Feedback on Player Experience During VR Gaming
Virtual, Augmented and Mixed RealityAbstractAs the interest in virtual reality (VR) technology as a game console has rapidly grown over the past few years, many new technologies are also being developed to further enhance the VR gaming experience. One of these technologies is haptic ...
Comments