Brooke Krajancich
Petr Kellnhofer
Skip Abstract Section
Skip Supplemental Material SectionAbstract
Virtual and augmented reality (VR/AR) displays strive to provide a resolution, framerate and field of view that matches the perceptual capabilities of the human visual system, all while constrained by limited compute budgets and transmission bandwidths of wearable computing systems. Foveated graphics techniques have emerged that could achieve these goals by exploiting the falloff of spatial acuity in the periphery of the visual field. However, considerably less attention has been given to temporal aspects of human vision, which also vary across the retina. This is in part due to limitations of current eccentricity-dependent models of the visual system. We introduce a new model, experimentally measuring and computationally fitting eccentricity-dependent critical flicker fusion thresholds jointly for both space and time. In this way, our model is unique in enabling the prediction of temporal information that is imperceptible for a certain spatial frequency, eccentricity, and range of luminance levels. We validate our model with an image quality user study, and use it to predict potential bandwidth savings 7X higher than those afforded by current spatial-only foveated models. As such, this work forms the enabling foundation for new temporally foveated graphics techniques.
Supplemental Material
- Rachel Albert, Anjul Patney, David Luebke, and Joohwan Kim. 2017. Latency requirements for foveated rendering in virtual reality. ACM Transactions on Applied Perception (TAP) 14, 4 (2017), 1--13.Google Scholar
Digital Library
- MR Ali and T Amir. 1991. Critical flicker frequency under monocular and binocular conditions. Perceptual and motor skills 72, 2 (1991), 383--386.Google Scholar
- Dale Allen and Robert F Hess. 1992. Is the visual field temporally homogeneous? Vision research 32, 6 (1992), 1075--1084.Google Scholar
- Elena Arabadzhiyska, Okan Tarhan Tursun, Karol Myszkowski, Hans-Peter Seidel, and Piotr Didyk. 2017. Saccade landing position prediction for gaze-contingent rendering. ACM Transactions on Graphics (TOG) 36, 4 (2017), 1--12.Google ScholarDigital Library
- Mark F Bradshaw and Brian J Rogers. 1999. Sensitivity to horizontal and vertical corrugations defined by binocular disparity. Vision research 39, 18 (1999), 3049--3056.Google Scholar
- David Carmel, Nilli Lavie, and Geraint Rees. 2006. Conscious awareness of flicker in humans involves frontal and parietal cortex. Current biology 16, 9 (2006), 907--911.Google Scholar
- Hanfeng Chen, Sung-Soo Kim, Sung-Hee Lee, Oh-Jae Kwon, and Jun-Ho Sung. 2005. Nonlinearity compensated smooth frame insertion for motion-blur reduction in LCD. In 2005 IEEE 7th Workshop on Multimedia Signal Processing. IEEE, 1--4.Google ScholarCross Ref
- Kajal T. Claypool and Mark Claypool. 2007. On frame rate and player performance in first person shooter games. Multimedia Systems 13, 1 (Sept. 2007), 3--17.Google ScholarDigital Library
- Mark Claypool and Kajal Claypool. 2009. Perspectives, Frame Rates and Resolutions: It's All in the Game. In Proc. Int. Conference on Foundations of Digital Games. Association for Computing Machinery, New York, NY, USA, 42--49.Google ScholarDigital Library
- Christine A Curcio and Kimberly A Allen. 1990. Topography of ganglion cells in human retina. Journal of comparative Neurology 300, 1 (1990), 5--25.Google ScholarCross Ref
- John G Daugman. 1985. Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical filters. JOSA A 2, 7 (1985), 1160--1169.Google ScholarCross Ref
- James Davis, Yi-Hsuan Hsieh, and Hung-Chi Lee. 2015. Humans perceive flicker artifacts at 500 Hz. Scientific Reports 5, 1 (Feb. 2015), 7861.Google ScholarCross Ref
- H de Lange Dzn. 1958. Research into the dynamic nature of the human fovea→cortex systems with intermittent and modulated light. I. Attenuation characteristics with white and colored light. Josa 48, 11 (1958), 777--784.Google ScholarCross Ref
- Michael F Deering. 1998. The limits of human vision. In 2nd International Immersive Projection Technology Workshop, Vol. 2. 1.Google Scholar
- Gyorgy Denes, Akshay Jindal, Aliaksei Mikhailiuk, and Rafał K. Mantiuk. 2020. A perceptual model of motion quality for rendering with adaptive refresh-rate and resolution. ACM Transactions on Graphics 39, 4 (July 2020), 133:133:1--133:133:17.Google ScholarDigital Library
- Gyorgy Denes, Kuba Maruszczyk, George Ash, and Rafał K. Mantiuk. 2019. Temporal Resolution Multiplexing: Exploiting the limitations of spatio-temporal vision for more efficient VR rendering. IEEE Transactions on Visualization and Computer Graphics 25, 5 (May 2019), 2072--2082.Google ScholarCross Ref
- Piotr Didyk, Elmar Eisemann, Tobias Ritschel, Karol Myszkowski, and Hans-Peter Seidel. 2010. Perceptually-motivated Real-time Temporal Upsampling of 3D Content for High-refresh-rate Displays. Computer Graphics Forum (Eurographics) 29, 2 (2010), 713--722.Google ScholarCross Ref
- Andrew T. Duchowski, Nathan Cournia, and Hunter A. Murphy. 2004. Gaze-Contingent Displays: A Review. Cyberpsychology & behavior 7 (2004), 621--34.Google Scholar
- Auria Eisen-Enosh, Nairouz Farah, Zvia Burgansky-Eliash, Idit Maharshak, Uri Polat, and Yossi Mandel. 2020. A dichoptic presentation device and a method for measuring binocular temporal function in the visual system. Experimental Eye Research 201 (2020), 108290.Google ScholarCross Ref
- Auria Eisen-Enosh, Nairouz Farah, Zvia Burgansky-Eliash, Uri Polat, and Yossi Mandel. 2017. Evaluation of critical flicker-fusion frequency measurement methods for the investigation of visual temporal resolution. Scientific reports 7, 1 (2017), 1--9.Google Scholar
- Sebastian Friston, Tobias Ritschel, and Anthony Steed. 2019. Perceptual Rasterization for Head-mounted Display Image Synthesis. ACM Trans. Graph. (Proc. SIGGRAPH 2019) 38, 4 (2019).Google Scholar
- Wilson S. Geisler and Jeffrey S. Perry. 1998. Real-time foveated multiresolution system for low-bandwidth video communication. In Human Vision and Electronic Imaging III, Vol. 3299. International Society for Optics and Photonics, 294--305.Google Scholar
- Brian Guenter, Mark Finch, Steven Drucker, Desney Tan, and John Snyder. 2012. Foveated 3D Graphics. ACM Trans. Graph. (SIGGRAPH Asia) (2012).Google Scholar
- E Hartmann, B Lachenmayr, and H Brettel. 1979. The peripheral critical flicker frequency. Vision Research 19, 9 (1979), 1019--1023.Google ScholarCross Ref
- Chia-Chiang Ho, Ja-Ling Wu, and Wen-Huang Cheng. 2005. A practical foveation-based rate-shaping mechanism for MPEG videos. IEEE transactions on circuits and systems for video technology 15, 11 (2005), 1365--1372.Google Scholar
- J-K Kamarainen, Ville Kyrki, and Heikki Kalviainen. 2006. Invariance properties of Gabor filter-based features-overview and applications. IEEE Transactions on image processing 15, 5 (2006), 1088--1099.Google ScholarDigital Library
- Anton S. Kaplanyan, Anton Sochenov, Thomas Leimkühler, Mikhail Okunev, Todd Goodall, and Gizem Rufo. 2019. DeepFovea: Neural Reconstruction for Foveated Rendering and Video Compression Using Learned Statistics of Natural Videos. ACM Trans. Graph. 38, 6, Article 212 (Nov. 2019), 13 pages.Google ScholarDigital Library
- D. H. Kelly. 1979. Motion and vision. II. Stabilized spatio-temporal threshold surface. JOSA 69, 10 (Oct. 1979), 1340--1349.Google ScholarCross Ref
- Jonghyun Kim, Youngmo Jeong, Michael Stengel, Kaan Akşit, Rachel Albert, Ben Boudaoud, Trey Greer, Joohwan Kim, Ward Lopes, Zander Majercik, et al. 2019. Foveated AR: dynamically-foveated augmented reality display. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1--15.Google ScholarDigital Library
- JJ Koenderink and AJ Van Doorn. 1978. Visual detection of spatial contrast; influence of location in the visual field, target extent and illuminance level. Biological Cybernetics 30, 3 (1978), 157--167.Google ScholarCross Ref
- Jan J Koenderink, Maarten A Bouman, Albert E Bueno de Mesquita, and Sybe Slappendel. 1978c. Perimetry of contrast detection thresholds of moving spatial sine wave patterns. II. The far peripheral visual field (eccentricity 0°-50°). JOSA 68, 6 (1978), 850--854.Google ScholarCross Ref
- Jan J Koenderink, Maarten A Bouman, Albert E Bueno de Mesquita, and Sybe Slappendel. 1978b. Perimetry of contrast detection thresholds of moving spatial sine wave patterns. III. The target extent as a sensitivity controlling parameter. JOSA 68, 6 (1978), 854--860.Google ScholarCross Ref
- Jan J Koenderink, Maarten A Bouman, Albert E Bueno de Mesquita, and Sybe Slappendel. 1978a. Perimetry of contrast detection thresholds of moving spatial sine wave patterns. IV. The influence of the mean retinal illuminance. JOSA 68, 6 (1978), 860--865.Google ScholarCross Ref
- George Alex Koulieris, Kaan Akşit, Michael Stengel, Rafał K Mantiuk, Katerina Mania, and Christian Richardt. 2019. Near-eye display and tracking technologies for virtual and augmented reality. In Computer Graphics Forum, Vol. 38. Wiley Online Library, 493--519.Google Scholar
- Brooke Krajancich, Petr Kellnhofer, and Gordon Wetzstein. 2020. Optimizing depth perception in virtual and augmented reality through gaze-contingent stereo rendering. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1--10.Google ScholarDigital Library
- Zhi Li, Anne Aaron, Ioannis Katsavounidis, Anush Moorthy, and Megha Manohara. 2016. Toward a practical perceptual video quality metric. The Netflix Tech Blog 6, 2 (2016).Google Scholar
- David Luebke and Benjamin Hallen. 2001. Perceptually driven simplification for interactive rendering. In Proceedings of the 12th Eurographics conference on Rendering (EGWR'01). Eurographics Association, Goslar, DEU, 223--234.Google ScholarDigital Library
- S Marĉelja. 1980. Mathematical description of the responses of simple cortical cells. JOSA 70, 11 (1980), 1297--1300.Google ScholarCross Ref
- Michael Mauderer, Simone Conte, Miguel A Nacenta, and Dhanraj Vishwanath. 2014. Depth perception with gaze-contingent depth of field. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 217--226.Google ScholarDigital Library
- John D. McCarthy, M. Angela Sasse, and Dimitrios Miras. 2004. Sharp or smooth? comparing the effects of quantization vs. frame rate for streamed video. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '04). Association for Computing Machinery, New York, NY, USA, 535--542.Google ScholarDigital Library
- Arian Mehrfard, Javad Fotouhi, Giacomo Taylor, Tess Forster, Nassir Navab, and Bernhard Fuerst. 2019. A comparative analysis of virtual reality head-mounted display systems. arXiv preprint arXiv:1912.02913 (2019).Google Scholar
- Hunter Murphy and Andrew T. Duchowski. 2001. Gaze-Contingent Level Of Detail Rendering. (2001).Google Scholar
- Toshikazu Ohshima, Hiroyuki Yamamoto, and Hideyuki Tamura. 1996. Gaze-directed adaptive rendering for interacting with virtual space. In Proceedings of the IEEE 1996 Virtual Reality Annual International Symposium. IEEE, 103--110.Google ScholarCross Ref
- Nitish Padmanaban, Robert Konrad, Tal Stramer, Emily A Cooper, and Gordon Wetzstein. 2017. Optimizing virtual reality for all users through gaze-contingent and adaptive focus displays. Proceedings of the National Academy of Sciences 114, 9 (2017), 2183--2188.Google ScholarCross Ref
- Anjul Patney, Marco Salvi, Joohwan Kim, Anton Kaplanyan, Chris Wyman, Nir Benty, David Luebke, and Aaron Lefohn. 2016. Towards foveated rendering for gaze-tracked virtual reality. ACM Transactions on Graphics (TOG) 35, 6 (Nov. 2016), 179:1--179:12.Google ScholarDigital Library
- Jonathan W Peirce. 2007. PsychoPy-psychophysics software in Python. Journal of neuroscience methods 162, 1-2 (2007), 8--13.Google ScholarCross Ref
- Denis G Pelli and Peter Bex. 2013. Measuring contrast sensitivity. Vision research 90 (2013), 10--14.Google Scholar
- JG Robson. 1993. Contrast sensitivity: One hundred years of clinical measurement in Proceedings of the Retina Research Foundation Symposia, Vol. 5 Eds R. Shapley, D. Man-Kit Lam.Google Scholar
- John G Robson. 1966. Spatial and temporal contrast-sensitivity functions of the visual system. Josa 56, 8 (1966), 1141--1142.Google ScholarCross Ref
- J. Gordon Robson and Norma Graham. 1981. Probability summation and regional variation in contrast sensitivity across the visual field. Vision research 21, 3 (1981), 409--418.Google Scholar
- Manuel Romero-Gómez, Juan Córdoba, Rodrigo Jover, Juan A Del Olmo, Marta Ramírez, Ramón Rey, Enrique De Madaria, Carmina Montoliu, David Nuñez, Montse Flavia, et al. 2007. Value of the critical flicker frequency in patients with minimal hepatic encephalopathy. Hepatology 45, 4 (2007), 879--885.Google ScholarCross Ref
- Ruth Rosenholtz. 2016. Capabilities and limitations of peripheral vision. Annual Review of Vision Science 2 (2016), 437--457.Google ScholarCross Ref
- Jyrki Rovamo and Antti Raninen. 1984. Critical flicker frequency and M-scaling of stimulus size and retinal illuminance. Vision research 24, 10 (1984), 1127--1131.Google Scholar
- Jyrki Rovamo and Antti Raninen. 1988. Critical flicker frequency as a function of stimulus area and luminance at various eccentricities in human cone vision: a revision of Granit-Harper and Ferry-Porter laws. Vision research 28, 7 (1988), 785--790.Google Scholar
- Theodore C. Ruch and John F. Fulton. 1960. Medical Physiology and Biophysics. Journal of Medical Education 35 (1960), 1067. Issue 11.Google Scholar
- Daniel Scherzer, Lei Yang, Oliver Mattausch, Diego Nehab, Pedro V. Sander, Michael Wimmer, and Elmar Eisemann. 2012. Temporal Coherence Methods in Real-Time Rendering. Computer Graphics Forum 31, 8 (2012), 2378--2408.Google ScholarDigital Library
- Kalpana Seshadrinathan, Rajiv Soundararajan, Alan Conrad Bovik, and Lawrence K Cormack. 2010. Study of subjective and objective quality assessment of video. IEEE transactions on Image Processing 19, 6 (2010), 1427--1441.Google Scholar
- Jie Shen, Christopher A Clark, P Sarita Soni, and Larry N Thibos. 2010. Peripheral refraction with and without contact lens correction. Optometry and vision science: official publication of the American Academy of Optometry 87, 9 (2010), 642.Google Scholar
- Raunak Sinha, Mrinalini Hoon, Jacob Baudin, Haruhisa Okawa, Rachel OL Wong, and Fred Rieke. 2017. Cellular and circuit mechanisms shaping the perceptual properties of the primate fovea. Cell 168, 3 (2017), 413--426.Google ScholarCross Ref
- Jan P Springer, Stephan Beck, Felix Weiszig, Dirk Reiners, and Bernd Froehlich. 2007. Multi-frame rate rendering and display. In 2007 IEEE Virtual Reality Conference. IEEE, 195--202.Google ScholarCross Ref
- Philip A Stanley and A Kelvin Davies. 1995. The effect of field of view size on steady-state pupil diameter. Ophthalmic and Physiological Optics 15, 6 (1995), 601--603.Google ScholarCross Ref
- Qi Sun, Fu-Chung Huang, Joohwan Kim, Li-Yi Wei, David Luebke, and Arie Kaufman. 2017. Perceptually-guided foveation for light field displays. ACM Transactions on Graphics 36, 6 (Nov. 2017), 192:1--192:13.Google ScholarDigital Library
- LN Thibos, DJ Walsh, and FE Cheney. 1987. Vision beyond the resolution limit: aliasing in the periphery. Vision Research 27, 12 (1987), 2193--2197.Google ScholarCross Ref
- Okan Tarhan Tursun, Elena Arabadzhiyska-Koleva, Marek Wernikowski, Radosław Mantiuk, Hans-Peter Seidel, Karol Myszkowski, and Piotr Didyk. 2019. Luminance-contrast-aware foveated rendering. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1--14.Google ScholarDigital Library
- Christopher W Tyler. 1987. Analysis of visual modulation sensitivity. III. Meridional variations in peripheral flicker sensitivity. JOSA A 4, 8 (1987), 1612--1619.Google ScholarCross Ref
- Christopher W Tyler and Russell D Hamer. 1990. Analysis of visual modulation sensitivity. IV. Validity of the Ferry-Porter law. JOSA A 7, 4 (1990), 743--758.Google ScholarCross Ref
- Christopher W Tyler and Russell D Hamer. 1993. Eccentricity and the Ferry-Porter law. JOSA A 10, 9 (1993), 2084--2087.Google ScholarCross Ref
- Floris L Van Nes and Maarten A Bouman. 1967. Spatial modulation transfer in the human eye. JOSA 57, 3 (1967), 401--406.Google ScholarCross Ref
- Zhou Wang, Ligang Lu, and Alan C Bovik. 2003. Foveation scalable video coding with automatic fixation selection. IEEE Transactions on Image Processing 12, 2 (2003), 243--254.Google ScholarDigital Library
- Andrew B Watson. 2018. The Field of View, the Field of Resolution, and the Field of Contrast Sensitivity. Journal of Perceptual Imaging 1, 1 (2018), 10505--1.Google ScholarCross Ref
- Andrew B Watson and Albert J Ahumada. 2016. The pyramid of visibility. Electronic Imaging 2016, 16 (2016), 1--6.Google Scholar
- Andrew B Watson and Denis G Pelli. 1983. QUEST: A Bayesian adaptive psychometric method. Perception & psychophysics 33, 2 (1983), 113--120.Google Scholar
- Thomas P Weldon, William E Higgins, and Dennis F Dunn. 1996. Efficient Gabor filter design for texture segmentation. Pattern recognition 29, 12 (1996), 2005--2015.Google Scholar
- Lei Xiao, Salah Nouri, Matt Chapman, Alexander Fix, Douglas Lanman, and Anton Kaplanyan. 2020. Neural Supersampling for Real-Time Rendering. ACM Trans. Graph. 39, 4, Article 142 (July 2020), 12 pages.Google ScholarDigital Library
Index Terms
- A perceptual model for eccentricity-dependent spatio-temporal flicker fusion and its applications to foveated graphics
Recommendations
Towards foveated rendering for gaze-tracked virtual reality
Foveated rendering synthesizes images with progressively less detail outside the eye fixation region, potentially unlocking significant speedups for wide field-of-view displays, such as head mounted displays, where target framerate and resolution is ...
Towards Attention–aware Foveated Rendering
Foveated graphics is a promising approach to solving the bandwidth challenges of immersive virtual and augmented reality displays by exploiting the falloff in spatial acuity in the periphery of the visual field. However, the perceptual models used in ...
Perceptually-based foveated virtual reality
SIGGRAPH '16: ACM SIGGRAPH 2016 Emerging TechnologiesHumans have two distinct vision systems: foveal and peripheral vision. Foveal vision is sharp and detailed, while peripheral vision lacks fidelity. The difference in characteristics of the two systems enable recently popular foveated rendering systems, ...
Comments