Jan Riemann
Martin Schmitz
Max Mühlhäuser
Skip Abstract Section
Skip Supplemental Material SectionAbstract
Tangible interaction has shown to be beneficial in a wide variety of scenarios since it provides more direct manipulation and haptic feedback. Further, inherently three-dimensional information is represented more naturally by a 3D object than by a flat picture on a screen. Yet, today's tangibles have often pre-defined form factors and limited input and output facilities. To overcome this issue, the combination of projection and depth cameras is used as a fast and flexible way of non-intrusively adding input and output to tangibles. However, tangibles are often quite small and hence the space for output and interaction on their surface is limited. Therefore, we propose FlowPut: an environment-aware framework that utilizes the space available on and around a tangible object for projected visual output. By means of an optimization-based layout approach, FlowPut considers the environment of the objects to avoid interference between projection and real-world objects. Moreover, we contribute an occlusion resilient object recognition and tracking for tangible objects based on their 3D model and a point-cloud based multi-touch detection, that allows sensing touches also on the side of a tangible. Flowput is validated through a series of technical experiments, a user study, and two example applications.
Supplemental Material
Available for Download
zip
riemann.zip (49.8 MB)
Supplemental movie, appendix, image and software files for, FlowPut: Environment-Aware Interactivity for Tangible 3D Objects
- Deepak Bandyopadhyay, Ramesh Raskar, and Henry Fuchs. 2001. Dynamic Shader Lamps: Painting on Movable Objects. In Proceedings of the IEEE and ACM International Symposium on Augmented Reality (ISAR '01) (ISAR '01). IEEE Computer Society, Washington, DC, USA, 207-. http://dl.acm.org/citation.cfm?id=582828.881324 Google Scholar
Digital Library
- Blaine Bell, Steven Feiner, and Tobias Höllerer. 2001. View Management for Virtual and Augmented Reality. In Proceedings of the 14th Annual ACM Symposium on User Interface Software and Technology (UIST '01). ACM, New York, NY, USA, 101--110. Google ScholarDigital Library
- Blaine A. Bell and Steven K. Feiner. 2000. Dynamic Space Management for User Interfaces. In Proceedings of the 13th Annual ACM Symposium on User Interface Software and Technology (UIST '00). ACM, New York, NY, USA, 239--248. Google ScholarDigital Library
- Eric Brockmeyer, Ivan Poupyrev, and Scott Hudson. 2013. PAPILLON: Designing Curved Display Surfaces with Printed Optics. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13). ACM, New York, NY, USA, 457--462. Google ScholarDigital Library
- Daniel Cotting and Markus Gross. 2006. Interactive environment-aware display bubbles. In Proceedings of the 19th annual ACM symposium on User interface software and technology (UIST '06). ACM, New York, NY, USA, 245--254. Google ScholarDigital Library
- Kiran Dandekar, Balasundar I Raju, and Mandayam a Srinivasan. 2003. 3-D Finite-Element Models of Human and Monkey Fingertips to Investigate the Mechanics of Tactile Sense. Journal of Biomechanical Engineering 125, 5 (2003), 682.Google ScholarCross Ref
- Barrett Ens, Eyal Ofek, Neil Bruce, and Pourang Irani. 2015. Spatial Constancy of Surface-Embedded Layouts Across Multiple Environments. In Proceedings of the 3rd ACM Symposium on Spatial User Interaction (SUI '15). ACM, New York, NY, USA, 65--68. Google ScholarDigital Library
- Markus Funk, Oliver Korn, and Albrecht Schmidt. 2014. An Augmented Workplace for Enabling User-defined Tangibles. In CHI '14 Extended Abstracts on Human Factors in Computing Systems (CHIEA '14). ACM, New York, NY, USA, 1285--1290. Google ScholarDigital Library
- Genki Furumi, Daisuke Sakamoto, and Takeo Igarashi. 2012. SnapRail: a tabletop user interface widget for addressing occlusion by physical objects. In Proceedings of the 2012 ACM international conference on Interactive tabletops and surfaces (ITS '12). ACM, New York, NY, USA, 193--196. Google ScholarDigital Library
- R. Gal, L. Shapira, E. Ofek, and P. Kohli. 2014. FLARE: Fast layout for augmented reality applications. In 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 207--212.Google Scholar
- Nan-Wei Gong, Jürgen Steimle, Simon Olberding, Steve Hodges, Nicholas Edward Gillian, Yoshihiro Kawahara, and Joseph A. Paradiso. 2014. PrintSense: A Versatile Sensing Technique to Support Multimodal Flexible Surface Interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 1407--1410. Google ScholarDigital Library
- J. M. Gottfried, R. Nair, S. Meister, C. S. Garbe, and D. Kondermann. 2014. Time of flight motion compensation revisited. In 2014 IEEE International Conference on Image Processing (ICIP). 5861--5865.Google Scholar
- Raphael Grasset, Tobias Langlotz, Denis Kalkofen, Markus Tatzgern, and Dieter Schmalstieg. 2012. Image-driven View Management for Augmented Reality Browsers. In Proceedings of the 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) (ISMAR '12). IEEE Computer Society, Washington, DC, USA, 177--186. Google ScholarDigital Library
- John Hardy and Jason Alexander. 2012. Toolkit Support for Interactive Projected Displays. In Proceedings of the 11th International Conference on Mobile and Ubiquitous Multimedia (MUM '12). ACM, New York, NY, USA, Article 42, 10 pages. Google ScholarDigital Library
- Björn Hartmann, Meredith Ringel Morris, Hrvoje Benko, and Andrew D. Wilson. 2010. Pictionaire: Supporting Collaborative Design Work by Integrating Physical and Digital Artifacts. In Proceedings of the 2010 ACM Conference on Computer Supported Cooperative Work (CSCW '10). ACM, New York, NY, USA, 421--424. Google ScholarDigital Library
- Hiroshi Ishii and Brygg Ullmer. 1997. Tangible Bits: Towards Seamless Interfaces Between People, Bits and Atoms. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI '97). ACM, New York, NY, USA, 234--241. Google ScholarDigital Library
- Shahram Izadi, Richard A. Newcombe, David Kim, Otmar Hilliges, David Molyneaux, Steve Hodges, Pushmeet Kohli, Jamie Shotton, Andrew J. Davison, and Andrew Fitzgibbon. 2011. KinectFusion: Real-time Dynamic 3D Surface Reconstruction and Interaction. In ACM SIGGRAPH 2011 Talks (SIGGRAPH '11). ACM, New York, NY, USA, Article 23, 1 pages. Google ScholarDigital Library
- B.R. Jones, R. Sodhi, R.H. Campbell, G. Garnett, and B.P. Bailey. 2010. Build your world and play in it: Interacting with surface particles on complex objects. In Mixed and Augmented Reality (ISMAR), 2010 9th IEEE International Symposium on. IEEE Computer Society, Washington, DC, USA, 165--174.Google Scholar
- Sasa Junuzovic, Kori Inkpen, Tom Blank, and Anoop Gupta. 2012. IllumiShare: Sharing Any Surface. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 1919--1928. Google ScholarDigital Library
- Yoshihiro Kawahara, Steve Hodges, Benjamin S. Cook, Cheng Zhang, and Gregory D. Abowd. 2013. Instant Inkjet Circuits: Lab-based Inkjet Printing to Support Rapid Prototyping of UbiComp Devices. In Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '13). ACM, New York, NY, USA, 363--372. Google ScholarDigital Library
- Mohammadreza Khalilbeigi, Roman Lissermann, Wolfgang Kleine, and Jürgen Steimle. 2012. FoldMe: Interacting with Double-sided Foldable Displays. In Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction (TEI '12). ACM, New York, NY, USA, 33--40. Google ScholarDigital Library
- Mohammadreza Khalilbeigi, Jürgen Steimle, Jan Riemann, Niloofar Dezfuli, Max Mühlhäuser, and James D. Hollan. 2013. ObjecTop: occlusion awareness of physical objects on interactive tabletops. In Proceedings of the 2013 ACM international conference on Interactive tabletops and surfaces (ITS '13). ACM, New York, NY, USA, 255--264. Google ScholarDigital Library
- Han-Jong Kim, Ju-Whan Kim, and Tek-Jin Nam. 2016. miniStudio: Designers' Tool for Prototyping Ubicomp Space with Interactive Miniature. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 213--224. Google ScholarDigital Library
- Florian Klompmaker, Karsten Nebe, and Alex Fast. 2012. dSensingNI: A Framework for Advanced Tangible Interaction Using a Depth Camera. In Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction (TEI '12). ACM, New York, NY, USA, 217--224. Google ScholarDigital Library
- Daisuke Kobayashi and Naoki Hashimoto. 2014. Spatial Augmented Reality by Using Depth-based Object Tracking. In ACM SIGGRAPH 2014 Posters (SIGGRAPH '14). ACM, New York, NY, USA, Article 33, 1 pages. Google ScholarDigital Library
- H. Koike, Y. Sato, and Y. Kobayashi. 2001. Integrating paper and digital information on EnhancedDesk: a method for realtime finger tracking on an augmented desk system. ACM Trans. Comput.-Hum. Interact. 8, 4 (Dec. 2001), 307--322. Google ScholarDigital Library
- Ryo Koizumi, Daisuke Kobayashi, and Naoki Hashimoto. 2015. Acceleration of Dynamic Spatial Augmented Reality System with a Depth Camera. In Proceedings of the 2015 International Conference on Cyberworlds (CW) (CW '15). IEEE Computer Society, Washington, DC, USA, 50--53. Google ScholarDigital Library
- Daisuke Kondo, Yuichi Shiwaku, and Ryugo Kijima. 2008. Free Form Projection Display and Application. In Proceedings of the 5th ACM/IEEE International Workshop on Projector Camera Systems (PROCAMS '08). ACM, New York, NY, USA, Article 8, 2 pages. Google ScholarDigital Library
- D. Leithinger and M. Haller. 2007. Improving Menu Interaction for Cluttered Tabletop Setups with User-Drawn Path Menus. In Horizontal Interactive Human-Computer Systems, 2007. TABLETOP '07. Second Annual IEEE International Workshop on. IEEE Computer Society, Washington, DC, USA, 121--128.Google Scholar
- Chunyuan Liao, Hao Tang, Qiong Liu, Patrick Chiu, and Francine Chen. 2010. FACT: fine-grained cross-media interaction with documents via a portable hybrid paper-laptop interface. In Proceedings of the international conference on Multimedia (MM '10). ACM, New York, NY, USA, 361--370. Google ScholarDigital Library
- Natan Linder and Pattie Maes. 2010. LuminAR: Portable Robotic Augmented Reality Interface Design and Prototype. In Adjunct Proceedings of the 23Nd Annual ACM Symposium on User Interface Software and Technology (UIST '10). ACM, New York, NY, USA, 395--396. Google ScholarDigital Library
- David Lindlbauer, Jörg Mueller, and Marc Alexa. 2017. Changing the Appearance of Real-World Objects By Modifying Their Surroundings. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 3954--3965. Google ScholarDigital Library
- H. Mitsuhara, Y. Yano, and T. Moriyama. 2010. Paper-top interface for supporting note-taking and its preliminary experiment. In Systems Man and Cybernetics (SMC), 2010 IEEE International Conference on. IEEE Computer Society, Washington, DC, USA, 3456--3462.Google Scholar
- Gaku Narita, Yoshihiro Watanabe, and Masatoshi Ishikawa. 2017. Dynamic Projection Mapping Onto Deforming Non-Rigid Surface Using Deformable Dot Cluster Marker. IEEE Transactions on Visualization and Computer Graphics 23, 3 (March 2017), 1235--1248. Google ScholarDigital Library
- Simon Olberding, Nan-Wei Gong, John Tiab, Joseph A. Paradiso, and Jürgen Steimle. 2013. A Cuttable Multi-touch Sensor. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13). ACM, New York, NY, USA, 245--254. Google ScholarDigital Library
- Simon Olberding, Michael Wessely, and Jürgen Steimle. 2014. PrintScreen: Fabricating Highly Customizable Thin-film Touch-displays. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14). ACM, New York, NY, USA, 281--290. Google ScholarDigital Library
- Makoto Ono, Buntarou Shizuki, and Jiro Tanaka. 2013. Touch 8 Activate: Adding Interactivity to Existing Objects Using Active Acoustic Sensing. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13). ACM, New York, NY, USA, 31--40. Google ScholarDigital Library
- Jason Orlosky, Kiyoshi Kiyokawa, and Haruo Takemura. 2013. Dynamic Text Management for See-through Wearable and Heads-up Display Systems. In Proceedings of the 2013 International Conference on Intelligent User Interfaces (IUI '13). ACM, New York, NY, USA, 363--370. Google ScholarDigital Library
- Ramesh Raskar, Greg Welch, Kok-Lim Low, and Deepak Bandyopadhyay. 2001. Shader Lamps: Animating Real Objects With Image-Based Illumination. In Rendering Techniques 2001, StevenJ. Gortler and Karol Myszkowski (Eds.). Springer Vienna, Vienna, Austria, 89--102. Google ScholarDigital Library
- Jan Riemann, Mohammadreza Khalilbeigi, Niloofar Dezfuli, and Max Mühlhäuser. 2015. StackTop: Hybrid Physical-Digital Stacking on Interactive Tabletops. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (CHIEA '15). ACM, New York, NY, USA, 1127--1132. Google ScholarDigital Library
- Jan Riemann, Mohammadreza Khalilbeigi, and Max Mühlhäuser. 2015. In-Situ Occlusion Resolution for Hybrid Tabletop Environments. In Human-Computer Interaction - INTERACT 2015, Julio Abascal, Simone Barbosa, Mirko Fetter, Tom Gross, Philippe Palanque, and Marco Winckler (Eds.). Lecture Notes in Computer Science, Vol. 9298. Springer International Publishing, Cham, Switzerland, 278--295. Google ScholarDigital Library
- Jan Riemann, Mohammadreza Khalilbeigi, Martin Schmitz, Sebastian Doeweling, Florian Müller, and Max Mühlhäuser. 2016. FreeTop: Finding Free Spots for Projective Augmentation. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHIEA '16). ACM, New York, NY, USA, 1598--1606. Google ScholarDigital Library
- Radu Bogdan Rusu, Nico Blodow, and Michael Beetz. 2009. Fast Point Feature Histograms (FPFH) for 3D Registration. In Proceedings of the 2009 IEEE International Conference on Robotics and Automation (ICRA'09) IEEE Press, Piscataway, NJ, USA, 1848--1853. http://dl.acm.org/citation.cfm?id=1703435.1703733 Google ScholarDigital Library
- M. Sano, K. Matsumoto, B. H. Thomas, and H. Saito. 2015. Rubix: Dynamic Spatial Augmented Reality by Extraction of Plane Regions with a RGB-D Camera. In 2015 IEEE International Symposium on Mixed and Augmented Reality. 148--151. Google ScholarDigital Library
- Makoto Sato and Kaori Fujinami. 2014. Nonoverlapped View Management for Augmented Reality by Tabletop Projection. J. Vis. Lang. Comput. 25, 6 (Dec. 2014), 891--902. Google ScholarDigital Library
- Munehiko Sato, Ivan Poupyrev, and Chris Harrison. 2012. Touché: Enhancing Touch Interaction on Humans, Screens, Liquids, and Everyday Objects. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 483--492. Google ScholarDigital Library
- Valkyrie Savage, Ryan Schmidt, Tovi Grossman, George Fitzmaurice, and Björn Hartmann. 2014. A Series of Tubes: Adding Interactivity to 3D Prints Using Internal Pipes. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14). ACM, New York, NY, USA, 3--12. Google ScholarDigital Library
- Valkyrie Savage, Xiaohan Zhang, and Björn Hartmann. 2012. Midas: Fabricating Custom Capacitive Touch Sensors to Prototype Interactive Objects. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (UIST '12). ACM, New York, NY, USA, 579--588. Google ScholarDigital Library
- Martin Schmitz, Mohammadreza Khalilbeigi, Matthias Balwierz, Roman Lissermann, Max Mühlhäuser, and Jürgen Steimle. 2015. Capricate: A Fabrication Pipeline to Design and 3D Print Capacitive Touch Sensors for Interactive Objects. In Proceedings of the 28th Annual ACM Symposium on User Interface Software 8 Technology (UIST '15). ACM, New York, NY, USA, 253--258. Google ScholarDigital Library
- Thitirat Siriborvornratanakul and Masanori Sugimoto. 2008. Clutter-aware Adaptive Projection Inside a Dynamic Environment. In Proceedings of the 2008 ACM Symposium on Virtual Reality Software and Technology (VRST '08). ACM, New York, NY, USA, 241--242. Google ScholarDigital Library
- Jürgen Steimle, Andreas Jordt, and Pattie Maes. 2013. Flexpad: Highly Flexible Bending Interactions for Projected Handheld Displays. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 237--246. Google ScholarDigital Library
- Kazuna Tsuboi, Yuji Oyamada, Maki Sugimoto, and Hideo Saito. 2013. 3D Object Surface Tracking Using Partial Shape Templates Trained from a Depth Camera for Spatial Augmented Reality Environments. In Proceedings of the Fourteenth Australasian User Interface Conference - Volume 139 (AUIC '13). Australian Computer Society, Inc., Darlinghurst, Australia, Australia, 125--126. http://dl.acm.org/citation.cfm?id=2525493.2525508 Google ScholarDigital Library
- John Underkoffler and Hiroshi Ishii. 1999. Urp: A Luminous-tangible Workbench for Urban Planning and Design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '99) ACM, New York, NY, USA, 386--393. Google ScholarDigital Library
- Pierre Wellner. 1993. Interacting with Paper on the DigitalDesk. Commun. ACM 36, 7 (July 1993), 87--96. Google ScholarDigital Library
- Karl Willis, Eric Brockmeyer, Scott Hudson, and Ivan Poupyrev. 2012. Printed Optics: 3D Printing of Embedded Optical Elements for Interactive Devices. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (UIST '12). ACM, New York, NY, USA, 589--598. Google ScholarDigital Library
- Andrew D. Wilson. 2005. PlayAnywhere: A Compact Interactive Tabletop Projection-vision System. In Proceedings of the 18th Annual ACM Symposium on User Interface Software and Technology (UIST '05). ACM, New York, NY, USA, 83--92. Google ScholarDigital Library
- A. D. Wilson. 2007. Depth-Sensing Video Cameras for 3D Tangible Tabletop Interaction. In Horizontal Interactive Human-Computer Systems, 2007. TABLETOP '07. Second Annual IEEE International Workshop on. 201--204.Google ScholarCross Ref
- Andrew D. Wilson. 2010. Using a Depth Camera As a Touch Sensor. In ACM International Conference on Interactive Tabletops and Surfaces (ITS '10). ACM, New York, NY, USA, 69--72. Google ScholarDigital Library
- Robert Xiao, Chris Harrison, and Scott E. Hudson. 2013. WorldKit: Rapid and Easy Creation of Ad-hoc Interactive Applications on Everyday Surfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 879--888. Google ScholarDigital Library
- Robert Xiao, Scott Hudson, and Chris Harrison. 2016. DIRECT: Making Touch Tracking on Ordinary Surfaces Practical with Hybrid Depth-Infrared Sensing. In Proceedings of the 2016 ACM on Interactive Surfaces and Spaces (ISS '16). ACM, New York, NY, USA, 85--94. Google ScholarDigital Library
- Zhengyou Zhang. 1994. Iterative point matching for registration of free-form curves and surfaces. International journal of computer vision 13 (October 1994), 119--152. https://www.microsoft.com/en-us/research/publication/iterative-point-matching-registration-free-form-curves-surfaces/ Google ScholarDigital Library
- Zhengyou Zhang. 2000. A Flexible New Technique for Camera Calibration. IEEE Trans. Pattern Anal. Mach. Intell. 22, 11 (Nov. 2000), 1330--1334. Google ScholarDigital Library
- Yi Zhou, Shuangjiu Xiao, Ning Tang, Zhiyong Wei, and Xu Chen. 2016. Pmomo: Projection Mapping on Movable 3D Object. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 781--790. Google ScholarDigital Library
Index Terms
- FlowPut: Environment-Aware Interactivity for Tangible 3D Objects
Recommendations
Making Public Displays Interactive Everywhere
As the number of large public displays increases, the need for interaction techniques to control them is emerging. One promising way to provide such interaction is through personal mobile devices. However, although much research has covered this topic, ...
3D Freehand Gestural Navigation for Interactive Public Displays
Users increasingly expect more-interactive experiences with public displays for applications including learning, gaming, urban visualization, and planning. However, user interaction with applications on public displays is challenging and often doesn't ...
Pen + touch = new tools
UIST '10: Proceedings of the 23nd annual ACM symposium on User interface software and technologyWe describe techniques for direct pen+touch input. We observe people's manual behaviors with physical paper and notebooks. These serve as the foundation for a prototype Microsoft Surface application, centered on note-taking and scrapbooking of ...
Comments