David B. Lindell
ABSTRACT
Emerging detector technologies are capable of ultrafast capture of single photons, enabling imaging at the speed of light. Not only can these detectors be used for imaging at essentially trillion frame-per-second rates, but combining them with computational algorithms has given rise to unprecedented new imaging capabilities. Computational time-resolved imaging has enabled new techniques for 3D imaging, light transport analysis, imaging around corners or behind occluders, and imaging through scattering media such as fog, murky water, or human tissue. With applications in autonomous navigation, robotic vision, human-computer interaction, and more, this is an area of rapidly growing interest. In this course, we provide an introduction to computational time-resolved imaging and single photon sensing with a focus on hardware, applications, and algorithms. We describe various types of emerging single-photon detectors, including single-photon avalanche diodes and avalanche photodiodes, which are among the most popular time-resolved detectors. Physically accurate models for these detectors are described, including modeling parameters and noise statistics used in most computational algorithms. From the application side, we discuss the use of ultrafast active illumination for 3D imaging and transient imaging, and we describe the state of the art in non-line-of-sight imaging, which requires modelling and inverting the propagation and scattering of light from a visible surface to a hidden object and back. We describe time-resolved computational algorithms used in each of these applications and offer insights on potential future directions.
Supplemental Material
Available for Download
- N. Abramson. Light-in-flight recording by holography. Opt. Lett., 3(4):121--123, Oct 1978. 5, 6Google Scholar
Cross Ref
- B. Ahn, A. Dave, A. Veeraraghavan, I. Gkioulekas, and A. C. Sankaranarayanan. Convolutional approximations to the general non-line-of-sight imaging operator. In Proc. ICCV, 2019. 7Google ScholarCross Ref
- V. Arellano, D. Gutierrez, and A. Jarabo. Fast back-projection for non-line of sight reconstruction. Opt. Express, 25(10):11574--11583, 2017. 8Google ScholarCross Ref
- M. Buttafava, J. Zeman, A. Tosi, K. Eliceiri, and A. Velten. Non-line-of-sight imaging using a time-gated single photon avalanche diode. Opt. Express, 23(16):20997--21011, 2015. 7Google ScholarCross Ref
- S. Chan, R. E. Warburton, G. Gariepy, J. Leach, and D. Faccio. Non-line-of-sight tracking of people at long range. Opt. Express, 25(9):10109--10117, 2017. 7Google ScholarCross Ref
- G. Gariepy, F. Tonolini, R. Henderson, J. Leach, and D. Faccio. Detection and tracking of moving objects hidden from view. Nat. Photonics, 10, Jan 2016. 7Google ScholarCross Ref
- I. Gkioulekas, A. Levin, F. Durand, and T. Zickler. Micron-scale light transport decomposition using inter-ferometry. ACM Trans. Graph., 34(4):37:1--37:14, July 2015. 5, 6, 7Google ScholarDigital Library
- O. Gupta, T. Willwacher, A. Velten, A. Veeraraghavan, and R. Raskar. Reconstruction of hidden 3d shapes using diffuse reflections. Opt. Express, 20(17):19096--19108, 2012. 8Google ScholarCross Ref
- F. Heide, S. Diamond, D. B. Lindell, and G. Wetzstein. Sub-picosecond photon-efficient 3d imaging using single-photon sensors. Sci. Rep., 8(1), 2018. 5, 7Google ScholarCross Ref
- F. Heide, M. B. Hullin, J. Gregson, and W. Heidrich. Low-budget transient imaging using photonic mixer devices. ACM Trans. Graph., 32(4):45:1--45:10, July 2013. 5, 6Google ScholarDigital Library
- F. Heide, M. O'Toole, K. Zang, D. B. Lindell, S. Diamond, and G. Wetzstein. Non-line-of-sight imaging with partial occluders and surface normals. ACM Trans. Graph., 38(3), 2019. 8Google ScholarDigital Library
- F. Heide, L. Xiao, W. Heidrich, and M. B. Hullin. Diffuse mirrors: 3d reconstruction from diffuse indirect illumination using inexpensive time-of-flight sensors. In Proc. CVPR, 2014. 7Google ScholarDigital Library
- J. Iseringhausen and M. B. Hullin. Non-line-of-sight reconstruction using efficient transient rendering. ACM Trans. Graph., 39(1):1--14, 2020. 8Google ScholarDigital Library
- A. Kadambi, R. Whyte, A. Bhandari, L. Streeter, C. Barsi, A. Dorrington, and R. Raskar. Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles. ACM Trans. Graph., 32(6):167, 2013. 5Google ScholarDigital Library
- A. Kirmani, T. Hutchison, J. Davis, and R. Raskar. Looking around the corner using transient imaging. In Proc. ICCV, 2009. 7Google ScholarCross Ref
- J. Klein, C. Peters, J. Martín, M. Laurenzis, and M. B. Hullin. Tracking objects outside the line of sight using 2d intensity images. Sci. Rep., 6:32491, 2016. 7Google ScholarCross Ref
- H. Kubo, S. Jayasuriya, T. Iwaguchi, T. Funatomi, Y. Mukaigawa, and S. G. Narasimhan. Programmable non-epipolar indirect light transport: Capture and analysis. IEEE Trans. Vis. Comput. Graph., 2019. 5Google ScholarCross Ref
- M. La Manna, F. Kine, E. Breitbach, J. Jackson, T. Sultan, and A. Velten. Error backprojection algorithms for non-line-of-sight imaging. IEEE Trans. Pattern Anal. Mach. Intell., 2018. 8Google Scholar
- D. B. Lindell, M. O'Toole, and G. Wetzstein. Single-photon 3d imaging with deep sensor fusion. ACM Trans. Graph., 37(4), 2018. 5Google ScholarDigital Library
- D. B. Lindell, M. O'Toole, and G. Wetzstein. Towards transient imaging at interactive rates with single-photon detectors. In Proc. ICCP, 2018. 5, 7Google ScholarCross Ref
- D. B. Lindell, G. Wetzstein, and M. O'Toole. Wave-based non-line-of-sight imaging using fast f-k migration. ACM Trans. Graph., 38(4):116, 2019. 5, 7, 8Google ScholarDigital Library
- X. Liu, S. Bauer, and A. Velten. Phasor field diffraction based reconstruction for fast non-line-of-sight imaging systems. Nat. Commun., 11(1):1--13, 2020. 8Google Scholar
- X. Liu, I. Guillén, M. La Manna, J. H. Nam, S. A. Reza, T. H. Le, A. Jarabo, D. Gutierrez, and A. Velten. Non-line-of-sight imaging using phasor-field virtual wave optics. Nature, 572(7771):620--623, 2019. 5Google ScholarCross Ref
- M. O'Toole, F. Heide, D. B. Lindell, K. Zang, S. Diamond, and G. Wetzstein. Reconstructing transient images from single-photon sensors. In Proc. CVPR, 2017. 5, 6, 7Google ScholarCross Ref
- M. O'Toole, D. B. Lindell, and G. Wetzstein. Real-time non-line-of-sight imaging. In ACM SIGGRAPH ETECH, 2018. 7Google ScholarDigital Library
- M. O'Toole, D. B. Lindell, and G. Wetzstein. Confocal non-line-of-sight imaging based on the light-cone transform. Nature, 555(7696):338, 2018. 5, 7, 8Google ScholarCross Ref
- A. K. Pediredla, M. Buttafava, A. Tosi, O. Cossairt, and A. Veeraraghavan. Reconstructing rooms using photon echoes: A plane based model and reconstruction algorithm for looking around the corner. In Proc. ICCP, 2017. 7Google ScholarCross Ref
- G. Satat, B. Heshmat, D. Raviv, and R. Raskar. All photons imaging through volumetric scattering. Sci. Rep., 6:33946, 2016. 5Google ScholarCross Ref
- G. Satat, M. Tancik, O. Gupta, B. Heshmat, and R. Raskar. Object classification through scattering media with deep learning on time resolved measurement. Opt. express, 25(15):17466--17479, 2017. 5Google ScholarCross Ref
- G. Satat, M. Tancik, and R. Raskar. Towards photography through realistic fog. In Proc. ICCP, 2018. 5Google ScholarCross Ref
- C. Saunders, J. Murray-Bruce, and V. K. Goyal. Computational periscopy with an ordinary digital camera. Nature, 565(7740):472, 2019. 7Google ScholarCross Ref
- B. Schwarz. Mapping the world in 3d. Nat. Photonics, 4(7):429--430, 2010. 5Google ScholarCross Ref
- D. Shin, F. Xu, D. Venkatraman, R. Lussana, F. Villa, F. Zappa, V. K. Goyal, F. N. C. Wong, and J. H. Shapiro. Photon-efficient imaging with a single-photon camera. Nat. Commun., 7(12046):1--7, 2016. 7Google Scholar
- O. Stavroudis. The optics of rays, wavefronts, and caustics, volume 38. Elsevier, 2012. 8Google Scholar
- S. Su, F. Heide, R. Swanson, J. Klein, C. Callenberg, M. Hullin, and W. Heidrich. Material classification using raw time-of-flight measurements. In Proc. CVPR, June 2016. 5Google ScholarCross Ref
- C.-Y. Tsai, K. N. Kutulakos, S. G. Narasimhan, and A. C. Sankaranarayanan. The geometry of first-returning photons for non-line-of-sight imaging. In Proc. CVPR, 2017. 5, 7Google ScholarCross Ref
- C.-Y. Tsai, A. Sankaranarayanan, and I. Gkioulekas. Beyond volumetric albedo---A surface optimization framework for non-line-of-sight imaging. In Proc. CVPR, 2019. 8Google ScholarCross Ref
- A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nat. Commun., 3, 2012. 5, 6, 7, 8Google Scholar
- A. Velten, D. Wu, A. Jarabo, B. Masia, C. Barsi, C. Joshi, E. Lawson, M. Bawendi, D. Gutierrez, and R. Raskar. Femto-photography: capturing and visualizing the propagation of light. ACM Trans. Graph., 32(4), 2013. 5Google ScholarDigital Library
- D. Wu, A. Velten, M. O'Toole, B. Masia, A. Agrawal, Q. Dai, and R. Raskar. Decomposing global light transport using time of flight imaging. Int. J. Comput. Vision, 107(2):123--138, Apr. 2014. 5Google ScholarDigital Library
- S. Xin, S. Nousias, K. N. Kutulakos, A. C. Sankaranarayanan, S. G. Narasimhan, and I. Gkioulekas. A theory of Fermat paths for non-line-of-sight shape reconstruction. In Proc. CVPR, 2019. 5, 8Google ScholarCross Ref
- F. Xu, G. Shulkind, C. Thrampoulidis, J. H. Shapiro, A. Torralba, F. N. Wong, and G. W. Wornell. Revealing hidden scenes by photon-efficient occlusion-based opportunistic active imaging. Opt. Express, 26(8):9945--9962, 2018. 7Google ScholarCross Ref
Recommendations
Protecting single-photon entanglement with imperfect single-photon source
Photon noiseless linear amplification (NLA) is a powerful way to protect the single photon or single-photon entanglement (SPE) from photon loss. However, current NLA protocols for single photon or SPE usually require the local auxiliary single photon, ...
Comments