Roger Boldu
Denys J.C. Matthies
Suranga Nanayakkara
Skip Abstract Section
Skip Supplemental Material SectionAbstract
People with visual impairments (PVI) experience simple tasks, such as grocery shopping, to be an essential difficulty. Although the recent emergence of AI-technology has been dramatically improving visual recognition capabilities, the application to the daily life of PVI is still complex and erroneous. For example, image recognition engines require a clear shot of the targeted object and a contextual understanding of the information the user requires. In this paper, we aimed to understand the PVI's needs and their pain points in the task of identifying grocery items. Following a user-centered design process, we iteratively
Supplemental Material
Available for Download
zip
boldu.zip (76.5 KB)
Supplemental movie, appendix, image and software files for, AiSee: An Assistive Wearable Device to Support Visually Impaired Grocery Shoppers
- Aira - Connecting you to real people instantly to simplify daily life. https://aira.io/Google Scholar
- Ariadne GPS | Mobility and map exploration for all. http://www.ariadnegps.eu/Google Scholar
- Be My Eyes - Bringing sight to blind and low-vision people. https://www.bemyeyes.com/Google Scholar
- BlindSquare. http://www.blindsquare.comGoogle Scholar
- TapTapSee - Blind and Visually Impaired Assistive Technology - powered by CloudSight.ai Image Recognition API. https://taptapseeapp.com/Google Scholar
- Vision API - Image Content Analysis Cloud Vision API Google Cloud. https://cloud.google.com/vision/Google Scholar
- ICD-11 - Mortality and Morbidity Statistics.Google Scholar
- Jeffrey P. Bigham, Chandrika Jayant, Andrew Miller, Brandyn White, and Tom Yeh. VizWiz::LocateIt - enabling blind people to locate objects in their environment. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops (2010-06). 65--72. https://doi.org/10.1109/CVPRW.2010.5543821 ISSN: 2160-7516.Google Scholar
- Roger Boldu, Alexandru Dancu, Denys J.C. Matthies, Thisum Buddhika, Shamane Siriwardhana, and Suranga Nanayakkara. Finger-Reader2.0: Designing and Evaluating a Wearable Finger-Worn Camera to Assist People with Visual Impairments while Shopping. In Proc. of IMWUT'18. ACM, 94.Google Scholar
- Roger Boldu, Alexandru Dancu, Denys J.C. Matthies, Pablo. Cascon, Shanaka Ransiri, and Suranga Nanayakkara. Thumb-In-Motion: Evaluating Thumb to Ring Microgestures for Athletic Activity. In Proceedings of the Symposium on Spatial User Interaction (SUI '18). ACM.Google Scholar
- Erin Brady, Meredith Ringel Morris, Yu Zhong, Samuel White, and Jeffrey P Bigham. Visual challenges in the everyday lives of blind people. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2117--2126.Google Scholar
- John Brooke et al. SUS-A quick and dirty usability scale. Usability evaluation in industry 189, 194 (1996), 4--7.Google Scholar
- Verena R Cimarolli, Kathrin Boerner, Mark Brennan-Ing, Joann P Reinhardt, and Amy Horowitz. Challenges faced by older adults with vision loss: a qualitative study with implications for rehabilitation. Clinical rehabilitation 26, 8 (2012), 748--757.Google Scholar
- Michael P Cutter and Roberto Manduchi. Towards mobile OCR: How to take a good picture of a document without sight. In Proceedings of the 2015 ACM Symposium on Document Engineering. ACM, 75--84.Google Scholar
- D. Dakopoulos and N. G. Bourbakis. Wearable Obstacle Avoidance Electronic Travel Aids for Blind: A Survey, Vol. 40. 25--35. https://doi.org/10.1109/TSMCC.2009.2021255Google Scholar
- HIMS International | Blaze ET. http://himsintl.com/product/blaze-et/Google Scholar
- Eyra. Horus. https://horus.tech.Google Scholar
- Umer Farooq and Jonathan Grudin. Human-computer integration. interactions 23, 6 (2016), 27--32.Google ScholarDigital Library
- KNFB Reader App features the best OCR. Turn print into speech or Braille instantly. iOS 3.0 now available. | KNFB Reader. https://knfbreader.com/Google Scholar
- Leah Findlater, Lee Stearns, Ruofei Du, Uran Oh, David Ross, Rama Chellappa, and Jon Froehlich. Supporting Everyday Activities for Persons with Visual Impairments Through Computer Vision-Augmented Touch. In Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility (2015) (ASSETS '15). ACM, 383--384. https://doi.org/10.1145/2700648.2811381Google Scholar
- Grace Sze-en Foo. Grocery Shopping Assistant for the Blind / Visually Impaired.. http://grozi.calit2.net/files/TIESGroZiSu09.pdf.Google Scholar
- James J Gibson. Observations on active touch. Psychological review 69, 6 (1962), 477.Google Scholar
- Google Brain. TensorFlow Release 1.2.1. https://goo.gl/WZqjLs.Google Scholar
- X. Guo, Y. Li, and H. Ling. LIME: Low-Light Image Enhancement via Illumination Map Estimation. IEEE Transactions on Image Processing 26, 2 (Feb. 2017), 982--993. https://doi.org/10.1109/TIP.2016.2639450 Conference Name: IEEE Transactions on Image Processing.Google ScholarDigital Library
- Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. VizWiz Grand Challenge: Answering Visual Questions from Blind People. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018-06). IEEE, 3608--3617. https://doi.org/10.1109/CVPR.2018.00380Google ScholarCross Ref
- Sandra G Hart and Lowell E Staveland. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Advances in psychology 52 (1988), 139--183.Google Scholar
- Step Hear. http://www.step-hear.com/Google Scholar
- Wilbert Jan Heeringa. Measuring Dialect Pronunciation Differences using Levenshtein Distance.Google Scholar
- Rabia Jafri, Syed Abid Ali, and Hamid R Arabnia. Computer Vision-based Object Recognition for the Visually Impaired Using Visual Tags. 7.Google Scholar
- Chandrika Jayant, Hanjie Ji, Samuel White, and Jeffrey P. Bigham. Supporting Blind Photography. In The Proceedings of the 13th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '11). ACM, New York, NY, USA, 203--210. https://doi.org/10.1145/2049536.2049573 event-place: Dundee, Scotland, UK.Google Scholar
- Hernisa Kacorri, Kris M. Kitani, Jeffrey P. Bigham, and Chieko Asakawa. People with Visual Impairment Training Personal Object Recognizers: Feasibility and Challenges. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (2017) (CHI '17). ACM, 5839--5849. https://doi.org/10.1145/3025453.3025899 event-place: Denver, Colorado, USA.Google Scholar
- Ryo Kawamura. RectLabel - Labeling images for object detection for MacOS. https://goo.gl/GVqq9H.Google Scholar
- Vladimir Kulyukin and Aliasgar Kutiyanawala. From ShopTalk to ShopMobile: vision-based barcode scanning with mobile phones for independent blind grocery shopping. In Proceedings of the 2010 Rehabilitation Engineering and Assistive Technology Society of North America Conference (RESNA 2010), Las Vegas, NV, Vol. 703. 1--5.Google Scholar
- Nicholas D Lane and Pete Warden. The deep (learning) transformation of mobile and embedded computing. Computer 51, 5 (2018), 12--16.Google ScholarCross Ref
- Patrick E Lanigan, Aaron M Paulos, Andrew W Williams, Dan Rossi, and Priya Narasimhan. Trinetra: Assistive Technologies for Grocery Shopping for the Blind.. In ISWC. 147--148.Google Scholar
- Sooyeon Lee, Chien Wen Yuan, Benjamin V. Hanrahan, Mary Beth Rosson, and John M. Carroll. Reaching Out: Investigating Different Modalities to Help People with Visual Impairments Acquire Items. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility - ASSETS '17 (2017). ACM Press, 389--390. https://doi.org/10.1145/3132525.3134817Google ScholarDigital Library
- Sooyeon Lee, Chien Wen Yuan, Benjamin V Hanrahan, Mary Beth Rosson, and John M Carroll. Reaching Out: Investigating Different Modalities to Help People with Visual I mpairments Acquire Items. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility. ACM, 389--390.Google Scholar
- Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott E. Reed, Cheng-Yang Fu, and Alexander C. Berg. SSD: Single Shot MultiBox Detector. CoRR abs/1512.02325. http://arxiv.org/abs/1512.02325Google Scholar
- Jack M Loomis and Susan J Lederman. Tactual perception. Handbook of perception and human performances 2 (1986), 2.Google Scholar
- Shiping Ma, Hongqiang Ma, Yuelei Xu, Shuai Li, Chao Lv, and Mingming Zhu. A Low-Light Sensor Image Enhancement Algorithm Based on HSI Color Model. Sensors 18, 10 (Oct. 2018), 3583. https://doi.org/10.3390/s18103583 Number: 10 Publisher: Multidisciplinary Digital Publishing Institute.Google Scholar
- Aipoly Fully Autonomous Markets. https://www.aipoly.com/Google Scholar
- Denys JC Matthies, Bodo Urban, Katrin Wolf, and Albrecht Schmidt. Reflexive Interaction: Extending the concept of Peripheral Interaction. In Proceedings of the 31st Australian Conference on Human-Computer-Interaction. 266--278.Google Scholar
- Microsoft. Seeing-AI. https://www.microsoft.com/en-us/seeing-ai/.Google Scholar
- John Nicholson, Vladimir Kulyukin, and Daniel Coster. ShopTalk: independent blind shopping through verbal route directions and barcode scans. The Open Rehabilitation Journal 2, 1 (2009), 11--23.Google ScholarCross Ref
- J.L. Pech-Pacheco, G. Cristobal, J. Chamorro-Martinez, and J. Fernandez-Valdivia. Diatom autofocusing in brightfield microscopy: a comparative study. In Proceedings 15th International Conference on Pattern Recognition. ICPR-2000 (2000), Vol. 3. IEEE Comput. Soc, 314--317. https://doi.org/10.1109/ICPR.2000.903548Google ScholarCross Ref
- Roy Shilkrot, Jochen Huber, Roger Boldu, Pattie Maes, and Suranga Nanayakkara. FingerReader: A Finger-Worn Assistive Augmentation. In Assistive Augmentation, Jochen Huber, Roy Shilkrot, Pattie Maes, and Suranga Nanayakkara (Eds.). Springer, Singapore, 151--175. https://doi.org/10.1007/978-981-10-6404-3_9Google Scholar
- Roy Shilkrot, Jochen Huber, Wong Meng Ee, Pattie Maes, and Suranga Chandima Nanayakkara. FingerReader: a wearable device to explore printed text on the go. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 2363--2372.Google Scholar
- Joan Sosa-Garcia and Francesca Odone. "Hands On" Visual Recognition for Visually Impaired Users. 10, 3 (2017), 8:1-8:30. https://doi.org/10.1145/3060056Google Scholar
- Lee Stearns, Ruofei Du, Uran Oh, Yumeng Wang, Leah Findlater, Rama Chellappa, and Jon E Froehlich. The Design and Preliminary Evaluation of a Finger-Mounted Camera and Feedback System to Enable Reading of Printed Text for the Blind.. In ECCV Workshops (3). 615--631.Google Scholar
- Brian Still and Kate Crane. Fundamentals of user-centered design: A practical approach. CRC Press.Google Scholar
- Sarit Szpiro, Yuhang Zhao, and Shiri Azenkot. Finding a store, searching for a product: a study of daily challenges of low vision people. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 61--72.Google Scholar
- Manoj V Thomas et al. iSee: Artificial Intelligence Based Android Application for Visually Impaired People. Journal of the Gujarat Research Society 21, 6 (2019), 200--208.Google Scholar
- Tzutalin. LabelImg, graphical image annotation tool on Windows and Linux. https://github.com/tzutalin/labelImg.Google Scholar
- Wayne Walls. Comparing image tagging services: Google Vision, Microsoft Cognitive Services, Amazon Rekognition and Clarifai. https://goo.gl/TVdzUR.Google Scholar
- Mark Weiser. The Computer for the 21 st Century. Scientific american 265, 3 (1991), 94--105.Google Scholar
- Samuel White, Hanjie Ji, and Jeffrey P. Bigham. EasySnap: Real-time Audio Feedback for Blind Photography. In Adjunct Proceedings of the 23Nd Annual ACM Symposium on User Interface Software and Technology (UIST '10). ACM, New York, NY, USA, 409--410. https://doi.org/10.1145/1866218.1866244 event-place: New York, New York, USA.Google Scholar
- Help People who are Blind or Partially Sighted. https://www.orcam.com/en/Google Scholar
- Jacob O Wobbrock and Julie A Kientz. Research contributions in human-computer interaction. interactions 23, 3 (2016), 38--44.Google ScholarDigital Library
- Katrin Wolf, Anja Naumann, Michael Rohs, and Jörg Müller. A taxonomy of microinteractions: Defining microgestures based on ergonomic and scenario-Dependent requirements. In IFIP conference on human-computer interaction. Springer, 559--575.Google Scholar
- Meng Ee Wong and Stacey S. K. Tan. Teaching the Benefits of Smart Phone Technology to Blind Consumers: Exploring the Potential of the iPhone. Journal of Visual Impairment & Blindness 106, 10 (2012), 646--650. https://doi.org/10.1177/0145482X1210601008Google ScholarCross Ref
- C. Yi, Y. Tian, and A. Arditi. Portable Camera-Based Assistive Text and Product Label Reading From Hand-Held Objects for Blind Persons. 19, 3 (2014), 808--817. https://doi.org/10.1109/TMECH.2013.2261083Google Scholar
- Z. Ying, G. Li, Y. Ren, R. Wang, and W. Wang. A New Low-Light Image Enhancement Algorithm Using Camera Response Model. In 2017 IEEE International Conference on Computer Vision Workshops (ICCVW). 3015--3022. https://doi.org/10.1109/ICCVW.2017.356 ISSN: 2473-9944.Google ScholarCross Ref
- Chien Wen Yuan, Benjamin V. Hanrahan, Sooyeon Lee, Mary Beth Rosson, and John M. Carroll. I Didn'T Know That You Knew I Knew: Collaborative Shopping Practices Between People with Visual Impairment and People with Vision, Vol. 1.118:1-118:18. Issue CSCW. https://doi.org/10.1145/3134753Google Scholar
- Tina Chien-Wen Yuan, Benjamin V. Hanrahan, Sooyeon Lee, Mary Beth Rosson, and John M. Carroll. I Didn't Know that You Knew I Knew: Collaborative Shopping Practices between People with Visual Impairment and People with Vision, Vol. 1. 118--118. https://doi.org/10.1145/3134753Google Scholar
- P. A. Zientara, S. Lee, G. H. Smith, R. Brenner, L. Itti, M. B. Rosson, J. M. Carroll, K. M. Irick, and V. Narayanan. Third Eye: A Shopping Assistant for the Visually Impaired, Vol. 50. 16--24. https://doi.org/10.1109/MC.2017.36Google Scholar
- VP Zinchenko and BF Lomov. The functions of hand and eye movements in the process of perception. Problems of Psychology 1, 2 (1960), 12--25.Google Scholar
Index Terms
- AiSee: An Assistive Wearable Device to Support Visually Impaired Grocery Shoppers
Recommendations
Identifying Visual Cues to Improve Independent IndoorNavigation for Blind Individuals
ASSETS '17: Proceedings of the 19th International ACM SIGACCESS Conference on Computers and AccessibilityThe idea of using technology to help those with visual impairments navigate has been studied extensively. However, most of these systems focus on getting the user from place to place, rather than helping the person get a better sense and intuition of ...
Usability and accessibility issues in the localization of assistive technology
Assets '06: Proceedings of the 8th international ACM SIGACCESS conference on Computers and accessibilityPeople with disabilities are faced with several barriers to computer usage. Various companies provide assistive software that makes computer usage possible for the population with disabilities. While increased awareness of disability issues has resulted ...
Lessons Learned from Designing, Deploying and Testing an Accessible BLE Beacon-based Wayfinding System in a Multi-Floor Indoor Environment
ASSETS '22: Proceedings of the 24th International ACM SIGACCESS Conference on Computers and AccessibilityIndoor wayfinding poses unique challenges for people with disabilities. Large, unfamiliar indoor environments can be difficult to navigate even for people with no disabilities. This paper presents a Bluetooth Low Energy (BLE) beacon-based indoor ...
Comments