Tahiya Chowdhury
Wendy Ju
ABSTRACT
Sensing activities at the city scale using big data can enable applications to improve the quality of citizen life. While there are approaches to sense the urban heartbeat using sound, vision, radio frequency (RF), and other sensors, capturing changes at urban scale using such sensing modalities is challenging. Due to the enormous amount of data they produce and the associated annotation and processing requirement, such data can be of limited use. In this paper, we present a vision-to-language modeling approach to capture patterns and transitions that occur in New York City from March 2020 to August 2020. We use the model on ~1 million street images captured by dashcams over 6 months. We then use the captions to train a language model based on Latent Dirichlet Allocation [4] and compare models from different periods using probabilistic distance measures. We observe distribution shifts in the model that correlate well with social distancing policies and are corroborated by different data sources, such as mobility traces. This language-based sensing introduces a new sensing modality to capture dynamics in the city with lower storage requirements and privacy concerns.
- [n.d.]. How the U.S. military buys location data from ordinary apps. https://www.vice.com/en/article/jgqm5x/us-military-location-data-xmode-locate-xGoogle Scholar
- 2021. Nexar. (2021). https://www.getnexar.comGoogle Scholar
- Tyler Atkinson, James Dolmas, Christoffer Koch, Evan F Koenig, Karel Mertens, Anthony Murphy, and Kei-Mu Yi. 2020. Mobility and engagement following the SARS-Cov-2 outbreak. (2020).Google Scholar
- David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet Allocation. J. Mach. Learn. Res. 3, null (March 2003), 993--1022.Google Scholar
- Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao. 2020. YOLOv4: Optimal Speed and Accuracy of Object Detection. CoRR abs/2004.10934 (2020). arXiv:2004.10934 https://arxiv.org/abs/2004.10934Google Scholar
- Anna Brown, Alexandra Chouldechova, Emily Putnam-Hornstein, Andrew Tobin, and Rhema Vaithianathan. 2019. Toward Algorithmic Accountability in Public Services: A Qualitative Study of Affected Community Perspectives on Algorithmic Decision-Making in Child Welfare Services. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). Association for Computing Machinery, New York, NY, USA, 1--12. Google ScholarDigital Library
- Serina Chang, Emma Pierson, Pang Wei Koh, Jaline Gerardin, Beth Redbird, David Grusky, and Jure Leskovec. 2021. Mobility network models of COVID-19 explain inequities and inform reopening. Nature 589, 7840 (2021), 82--87.Google Scholar
- Ben Charoenwong, Alan Kwan, and Vesa Pursiainen. 2020. Social connections with COVID-19--affected areas increase compliance with mobility restrictions. Science advances 6, 47 (2020), eabc3054.Google Scholar
- Marc N Conte, Matthew Gordon, Nicole Swartwood, Rachel Wilwerding, and Chu A Alex Yu. 2021. The causal effects of chronic air pollution on the intensity of COVID-19 disease: Some answers are blowing in the wind. medRxiv (2021).Google Scholar
- Marco Cristani, Alessio Del Bue, Vittorio Murino, Francesco Setti, and Alessandro Vinciarelli. 2020. The visual social distancing problem. IEEE Access 8 (2020), 126876--126886.Google ScholarCross Ref
- Yves-Alexandre De Montjoye, César A Hidalgo, Michel Verleysen, and Vincent D Blondel. 2013. Unique in the crowd: The privacy bounds of human mobility. Scientific reports 3, 1 (2013), 1--5.Google Scholar
- Maryam Farboodi, Gregor Jarosch, and Robert Shimer. 2021. Internal and external effects of social distancing in a pandemic. Journal of Economic Theory 196 (2021), 105293.Google ScholarCross Ref
- Song Gao, Jinmeng Rao, Yuhao Kang, Yunlei Liang, Jake Kruse, Dorte Dopfer, Ajay K Sethi, Juan Francisco Mandujano Reyes, Brian S Yandell, and Jonathan A Patz. 2020. Association of mobile phone location data indications of travel and stay-at-home mandates with COVID-19 infection rates in the US. JAMA network open 3, 9 (2020), e2020485--e2020485.Google Scholar
- Isha Ghodgaonkar, Subhankar Chakraborty, Vishnu Banna, Shane Allcroft, Mohammed Metwaly, Fischer Bordwell, Kohsuke Kimura, Xinxin Zhao, Abhinav Goel, Caleb Tung, Akhil Chinnakotla, Minghao Xue, Yung-Hsiang Lu, Mark Daniel Ward, Wei Zakharov, David S. Ebert, David M. Barbarash, and George K. Thiruvathukal. 2020. Analyzing Worldwide Social Distancing through Large-Scale Computer Vision. CoRR abs/2008.12363 (2020). arXiv:2008.12363 https://arxiv.org/abs/2008.12363Google Scholar
- Micah Hodosh, Peter Young, and Julia Hockenmaier. 2013. Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics. J. Artif. Int. Res. 47, 1 (May 2013), 853--899.Google ScholarDigital Library
- Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé, Miro Dudik, and Hanna Wallach. 2019. Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). Association for Computing Machinery, New York, NY, USA, 1--16. Google ScholarDigital Library
- MD. Zakir Hossain, Ferdous Sohel, Mohd Fairuz Shiratuddin, and Hamid Laga. 2019. A Comprehensive Survey of Deep Learning for Image Captioning. ACM Comput. Surv., Article 118 (Feb. 2019), 36 pages.Google ScholarDigital Library
- Xiao Huang, Zhenlong Li, Junyu Lu, Sicheng Wang, Hanxue Wei, and Baixu Chen. 2020. Time-series clustering for home dwell time during COVID-19: what can we learn from it? ISPRS International Journal of Geo-Information 9, 11 (2020), 675.Google ScholarCross Ref
- Cornelia Ilin, Sebastien Annan-Phan, Xiao Hui Tai, Shikhar Mehra, Solomon Hsiang, and Joshua E Blumenstock. 2021. Public mobility data enables covid-19 forecasting and management at local and global scales. Scientific reports 11, 1 (2021), 1--11.Google Scholar
- Wendy Ju, Sharon Yavo-Ayalon, Ilan Mandel, Federico Saldarini, Natalie Friedman, Srinath Sibi, J. D. Zamfirescu-Pereira, and Jorge Ortiz. 2020. Tracking Urban Mobility and Occupancy under Social Distancing Policy. Digit. Gov.: Res. Pract. 1, 4, Article 32 (Oct. 2020), 12 pages. Google ScholarDigital Library
- Farzam Kharvari and William O'Brien. 2020. C-HVAC: A Practical Tool for Assessing Ventilation Capacity for HVAC Systems during the COVID-19 Pandemic. In Proceedings of the 7th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation (Virtual Event, Japan) (BuildSys '20). Association for Computing Machinery, New York, NY, USA, 338--339. Google ScholarDigital Library
- Matthew R Lamb, Sasikiran Kandula, and Jeffrey Shaman. 2021. Differential COVID-19 case positivity in New York City neighborhoods: Socioeconomic factors and mobility. Influenza and Other Respiratory Viruses 15, 2 (2021), 209--217.Google ScholarCross Ref
- Yiqun Ma, Sen Pei, Jeffrey Shaman, Robert Dubrow, and Kai Chen. 2021. Role of meteorological factors in the transmission of SARS-CoV-2 in the United States. Nature Communications 12, 1 (2021), 1--9.Google ScholarCross Ref
- Praveen Manoharan, Srinarayana Nagarathinam, Arunchandar Vasan, and Venkata Ramakrishna P. 2020. Keep It Open, Keep It Safe: Maximizing Space Utilization in Intelligent Bio-Safe Buildings. In Proceedings of the 7th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation (Virtual Event, Japan) (BuildSys '20). Association for Computing Machinery, New York, NY, USA, 100--109. Google ScholarDigital Library
- Kingsley Nweye and Zoltan Nagy. 2020. Impact of COVID-19 on Academic Campus Energy Use. In Proceedings of the 7th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation (Virtual Event, Japan) (BuildSys '20). Association for Computing Machinery, New York, NY, USA, 322--323. Google ScholarDigital Library
- Adi Robertson. 2021. Google bans tracking tool that sold users' location data. https://www.theverge.com/2021/8/12/22621685/google-ban-safegraph-android-user-data-location-trackingGoogle Scholar
- Safegraph. 2021. Places Pattern. (2021). https://docs.safegraph.com/docs/Google Scholar
- Andres Sevtsuk, Annie Hudson, Dylan Halpern, Rounaq Basu, Kloe Ng, and Jorrit de Jong. 2021. The impact of COVID-19 on trips to urban amenities: Examining travel behavior changes in Somerville, MA. PloS one 16, 9 (2021), e0252794.Google Scholar
- Justin Sherman. [n.d.]. Data Brokers and Sensitive Data on US Individuals. ([n.d.]).Google Scholar
- Douglas G. Simpson. 1987. Minimum Hellinger distance estimation for the analysis of count data. J. Amer. Statist. Assoc. 82, 399 (Sept. 1987), 802--807. Funding Information: * Douglas G. Simpson is Assistant Professor, Department of Statistics and Institute for Environmental Studies, University of Illinois, Champaign, IL 61820. This work was partially supported by National Science Foundation Contract DMS 84O0602. The author thanks B'arry H. Mar-golin for stimulating this research and Raymond J. Carroll and David Ruppert for helpful suggestions. Google ScholarCross Ref
- Michael Veale, Max Van Kleek, and Reuben Binns. 2018. Fairness and Accountability Design Needs for Algorithmic Support in High-Stakes Public Sector Decision-Making. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI '18). Association for Computing Machinery, New York, NY, USA, 1--14. Google ScholarDigital Library
- Joakim A Weill, Matthieu Stigler, Olivier Deschenes, and Michael R Springborn. 2020. Social distancing responses to COVID-19 emergency declarations strongly differentiated by income. Proceedings of the National Academy of Sciences 117, 33 (2020), 19658--19660.Google ScholarCross Ref
- Vispute S. Espinosa V. Wellenius, G.A. 2021. Impacts of social distancing policies on mobility and COVID-19 case growth in the US. Nature Communication (2021). Google ScholarCross Ref
- Yiting Xia, Ying Zhang, Zhizhen Zhong, Guanqing Yan, Chiunlin Lim, Satyajeet Singh Ahuja, Soshant Bali, Alexander Nikolaidis, Kimia Ghobadi, and Manya Ghobadi. 2021. A Social Network Under Social Distancing: Risk-Driven Backbone Management During COVID-19 and Beyond.. In NSDI. 217--231.Google Scholar
- Dongfang Yang, Ekim Yurtsever, Vishnu Renganathan, Keith A. Redmill, and Ümit Özgüner. 2021. A Vision-Based Social Distancing and Critical Density Detection System for COVID-19. Sensors 21, 13 (2021). Google ScholarCross Ref
Index Terms
- Tracking urban heartbeat and policy compliance through vision and language-based sensing
Recommendations
Towards sensing urban-scale COVID-19 policy compliance in new york city
BuildSys '21: Proceedings of the 8th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and TransportationBig data on the urban scale can enable many applications for improving city life and provide a more holistic understanding of urban life to researchers. While there are approaches to sense and model urban occupant behaviors using sound, radio frequency, ...
Urban Computing: Concepts, Methodologies, and Applications
Special Section on Urban ComputingUrbanization's rapid progress has modernized many people's lives but also engendered big issues, such as traffic congestion, energy consumption, and pollution. Urban computing aims to tackle these issues by using the data that has been generated in ...
An ubiquitous service-oriented architecture for urban sensing
CARE/AVSA@AAMAS'14: Proceedings of the 2014 International Conference on Agent Technology for Intelligent Mobile Services and Smart Societies - 2014 Collaborative Agents, Research and Development, and 2014 Agents, Virtual Societies and AnalyticsIn the transformation from traditional to smart cities, there is an increasing trend around the world towards intelligent dynamic infrastructures that provide citizens with new services that can improve their quality of life and fulfill the criteria of ...
Comments