Morgan Dixon
Gierad Laput
ABSTRACT
Pixel-based methods offer unique potential for modifying existing interfaces independent of their underlying implementation. Prior work has demonstrated a variety of modifications to existing interfaces, including accessibility enhancements, interface language translation, testing frameworks, and interaction techniques. But pixel-based methods have also been limited in their understanding of the interface and therefore the complexity of modifications they can support. This work examines deeper pixel-level understanding of widgets and the resulting capabilities of pixel-based runtime enhancements. Specifically, we present three new sets of methods: methods for pixel-based modeling of widgets in multiple states, methods for managing the combinatorial complexity that arises in creating a multitude of runtime enhancements, and methods for styling runtime enhancements to preserve consistency with the design of an existing interface. We validate our methods through an implementation of Moscovich et al.'s Sliding Widgets, a novel runtime enhancement that could not have been implemented with prior pixel-based methods.
Supplemental Material
- Albinsson, P.-A. and Zhai, S. High Precision Touch Screen Interaction. CHI 2003, 105--112. Google Scholar
Digital Library
- Banovic, N., Grossman, T., Matejka, J., and Fitzmaurice, G. Waken: Reverse Engineering Usage Information and Interface Structure from Software Videos. UIST 2012, 83--92. Google ScholarDigital Library
- Baudisch, P. and Chu, G. Back-of-Device Interaction Allows Creating Very Small Touch Devices. CHI 2009, 1923--1932. Google ScholarDigital Library
- Benko, H., Wilson, A.D., and Baudisch, P. Precise Selection Techniques for Multi-Touch Screens. CHI 2006, 1263--1272. Google ScholarDigital Library
- Bolin, M., Webber, M., Rha, P., Wilson, T., and Miller, R.C. Automation andCustomizationofRenderedWebPages.UIST2005, 163--172. Google ScholarDigital Library
- Chang, T.-H., Yeh, T., and Miller, R. Associating the Visual Representation of User Interfaces with Their Internal Structures and Metadata. UIST 2011, 245--254. Google ScholarDigital Library
- Chang, T.-H., Yeh, T., and Miller, R.C. GUI Testing Using Computer Vision. CHI 2010, 1535--1544. Google ScholarDigital Library
- Dixon, M., Fogarty, J., and Wobbrock, J. A General-Purpose Target-Aware Pointing Enhancement Using Pixel-Level Analysis of Graphical Interfaces. CHI 2012, 3167--3176. Google ScholarDigital Library
- Dixon, M. and Fogarty, J. Prefab Layers and Prefab Annotations: Extensible Pixel-Based Interpretation of Graphical Interfaces. In Preparation.Google Scholar
- Dixon, M. and Fogarty, J. Prefab : Implementing Advanced Behaviors Using Pixel-Based Reverse Engineering of Interface Structure. CHI 2010, 1525--1534. Google ScholarDigital Library
- Dixon, M., Leventhal, D., and Fogarty, J. Content and Hierarchy in Pixel-Based Methods for Reverse Engineering Interface Structure. CHI 2011, 969--978. Google ScholarDigital Library
- Eagan, J.R., Beaudouin-Lafon, M., and Mackay, W.E. Cracking the Cocoa Nut: User Interface Programming at Runtime. UIST 2011, 225--234. Google ScholarDigital Library
- Edwards, W.K., Hudson, S.E., Marinacci, J., Rodenstein, R., Rodriguez, T., and Smith, I. Systematic Output Modification in a 2D User Interface Toolkit. UIST 1997, 151--158. Google ScholarDigital Library
- Fujima, J., Lunzer, A., Hornbæk, K., and Tanaka, Y. Clip, Connect, Clone: Combining Application Elements to Build Custom Interfaces for Information Access. UIST 2004, 175. Google ScholarDigital Library
- Gajos, K.Z., Wobbrock, J.O., and Weld, D.S. Automatically Generating User Interfaces Adapted To Users' Motor and Vision Capabilities. UIST 2007, 231--240. Google ScholarDigital Library
- Grossman, T. and Balakrishnan, R. The Bubble Cursor : Enhancing Target Acquisition by Dynamic Resizing of the Cursor's Activation Area. CHI 2005, 281--290. Google ScholarDigital Library
- Grossman, T., Matejka, J., and Fitzmaurice, G. Chronicle: Capture, Exploration, and Playback of Document Workflow Histories. UIST 2010, 143--152. Google ScholarDigital Library
- Hartmann, B., Wu, L., Collins, K., and Klemmer, S.R. Programming by a Sample: Rapidly Creating Web Applications with d.mix. UIST 2007, 241--250. Google ScholarDigital Library
- Hudson, S.E. and Smith, I. Supporting Dynamic Downloadable Appearances in an Extensible User Interface toolkit. UIST 1997, 159--168. Google ScholarDigital Library
- Hudson, S.E. and Tanaka, K. Providing Visually Rich Resizable Images for User Interface Components. UIST 2000, 227--235. Google ScholarDigital Library
- Hurst, A., Hudson, S.E., and Mankoff, J. Automatically Identifying Targets Users Interact with During Real World Tasks. IUI 2010, 11--20. Google ScholarDigital Library
- Kumar, R., Talton, J.O., Ahmad, S., and Klemmer, S.R. Bricolage : Example-Based Retargeting for Web Design. CHI 2011, 2197--2206. Google ScholarDigital Library
- Little, G., Lau, T.A., Cypher, A., Lin, J., Haber, E.M., and Kandogan, E. Koala: Capture, Share, Automate, Personalize Business Processes on the Web. CHI 2007, 943--952. Google ScholarDigital Library
- Moscovich, T. Contact Area Interaction with Sliding Widgets. UIST, (2009), 13--22. Google ScholarDigital Library
- Nichols, J., Hua, Z., and Barton, J. Highlight: A System for Creating and Deploying Mobile Web Applications. UIST 2008, 249--258. Google ScholarDigital Library
- Nichols, J. and Lau, T. Mobilization by Demonstration: Using Traces to Re-author Existing Web Sites. IUI 2008, 149--160. Google ScholarDigital Library
- Nichols, J., Myers, B. a., and Litwack, K. Improving Automatic Interface Generation with Smart Templates. IUI 2004, 286--288. Google ScholarDigital Library
- Olsen, D.R., Hudson, S.E., Verratti, T., Heiner, J.M., and Phelps, M. Implementing Interface Attachments Based on Surface Representations. CHI 1999, 191--198. Google ScholarDigital Library
- Olsen, D.R., Taufer, T., and Fails, J.A. ScreenCrayons: Annotating Anything. UIST 2004, 165--174. Google ScholarDigital Library
- Olwal, A., Feiner, S., and Heyman, S. Rubbing and Tapping for Precise and Rapid Selection on Touch-Screen Displays. CHI 2008, 295--304. Google ScholarDigital Library
- Pongnumkul, S., Dontcheva, M., Li, W., Wang, J., Bourdev, L., Avidan, S., and Cohen, M.F. Pause-and-Play: Automatically Linking Screencast Video Tutorials with Applications. UIST 2011, 135--144. Google ScholarDigital Library
- Potter, R. Triggers: Guiding Automation with Pixels to Achieve Data Access. A. Cypher, eds. MIT Press.Google Scholar
- Savva, M., Kong, N., Chhajta, A., Fei-fei, L., Agrawala, M., and Heer, J. ReVision : Automated Classification , Analysis and Redesign of Chart Images. UIST 2011, 393--402. Google ScholarDigital Library
- St Amant, R., Lieberman, H., and Potter, R. Visual Generalization in Programming by Example. Communications of the ACM 43, 3 (2000), 107--114. Google ScholarDigital Library
- St Amant, R., Riedl, R., Ritter, F.E., and Reifers, A. Image Processing in Cognitive Models with SegMan. HCII 2005.Google Scholar
- Stuerzlinger, W., Chapuis, O., Phillips, D., and Roussel, N. User Interface Façades: Towards Fully Adaptable User Interfaces. UIST 2006, 309--318. Google ScholarDigital Library
- Tan, D.S., Meyers, B., and Czerwinski, M. WinCuts : Manipulating Arbitrary Window Regions for More Effective Use of Screen Space. CHI 2004, 1525--1528. Google ScholarDigital Library
- Vogel, D. and Baudisch, P. Shift : A Technique for Operating Pen-Based Interfaces Using Touch. CHI 2007, 657--666. Google ScholarDigital Library
- Waldner, M., Steinberger, M., Grasset, R., and Schmalstieg, D. Importance-Driven Compositing Window Management: CHI 2011, 959--968. Google ScholarDigital Library
- Weldon, J. and Shneidermanss, B. Improving the Accuracy of Touch Screens: An Experimental Evaluation of Three Strategies. CHI 1988, 27--32. Google ScholarDigital Library
- Wigdor, D., Forlines, C., Baudisch, P., Barnwell, J., and Shen, C. LucidTouch : A See-Through Mobile Device. CHI 2007, 269--278. Google ScholarDigital Library
- Yeh, T., Chang, T.-H., and Miller, R.C. Sikuli: Using GUI Screenshots for Search and Automation. UIST 2009, 183--194. Google ScholarDigital Library
- Yeh, T., Chang, T.-H., Xie, B., Walsh, G., Watkins, I., Wongsuphasawat, K., Huang, M., Davis, L.S., and Bederson, B.B. Creating Contextual Help for GUIs Using Screenshots. UIST 2011, 145--154. Google ScholarDigital Library
- Zettlemoyer, L.S. and St. Amant, R. A Visual Medium for Programmatic Control of Interactive Applications. CHI 1999, 199--206. Google ScholarDigital Library
- Zettlemoyer, L.S., Amant, R.S., and Dulberg, M.S. IBOTS : Agent Control Through the User Interface. IUI 1999. Google ScholarDigital Library
Index Terms
- Pixel-based methods for widget state and style in a runtime implementation of sliding widgets
Recommendations
Content and hierarchy in pixel-based methods for reverse engineering interface structure
CHI '11: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsThe rigidity and fragmentation of GUI toolkits are fundamentally limiting the progress and impact of interaction research. Pixel-based methods offer unique potential for addressing these challenges independent of the implementation of any particular ...
Prefab layers and prefab annotations: extensible pixel-based interpretation of graphical interfaces
UIST '14: Proceedings of the 27th annual ACM symposium on User interface software and technologyPixel-based methods have the potential to fundamentally change how we build graphical interfaces, but remain difficult to implement. We introduce a new toolkit for pixel based enhancements, focused on two areas of support. Prefab Layers helps developers ...
A general-purpose target-aware pointing enhancement using pixel-level analysis of graphical interfaces
CHI '12: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsWe present a general-purpose implementation of a target aware pointing technique, functional across an entire desktop and independent of application implementations. Specifically, we implement Grossman and Balakrishnan's Bubble Cursor, the fastest ...
Comments