From the web of data to a world of action
Introduction
From human readable web pages, to formal semantics of linked data and the emergent social semantics of tags and folksonomies, we routinely look to the web as both a source of information and a place to put data. However, the web is also a locus of action: users want to get things done, whether booking a hotel room, or editing an online spreadsheet.
The boundaries between web and desktop interaction are blurring. On the one hand, the traditional PC desktop is now inhabited by widgets such as the Mac Dashboard, web fast-download apps such as Java Web Start or Adobe Air, and expanded browser functionality such as Chrome. On the other hand, computation and applications that once were part of the desktop are now hosted on the web (for example word-processing with Google Docs), and various technologies enable web applications to function even when users have no connectivity to the internet (for example DojoX [33], Google Gears and the offline mode of HTML5 [75], [76]). Furthermore, in emerging markets such as India and China, this convergence will be total, as the sole computing experience for many will be through mobile devices and predominantly the web.
So far most of these web-like or web-based applications are separate, and web activity is glued together by the user, often through crude cutting and pasting between web applications. This separation is epitomised by Google Web Elements,1 which embed content in users’ web pages, but, with the exception of Google maps, are largely sealed from one another and their context; while considerably more functional than data feeds, in the end no more integrated than early web syndication. Embedded applications, such as those in Facebook or Google Widgets, are integrated more richly with their respective underlying platforms, but again largely firewalled from each other, preventing synergistic interactions.
However, there is an emerging need to offer greater support to users in performing web-based activity that cut across individual applications, and potentially to partially automate common tasks.
For over 20 years the dominant interface paradigm has been instrumental: populating the interface with virtual ‘things’ (documents, shapes, files as icons) that are made as transparent as possible and manipulated ‘directly’ by the user [67], [46]. However, the balance is changing and a level of ‘intelligent’, mediated interaction is becoming more accepted. This is partly because Moore's law means that it is easier to do more clever things, but more significantly because of the changing environment.
On the big-screen web (web on a desktop or laptop PC), this is largely due to the sheer size of data available on the web, so that Google search or Amazon suggestions become acceptable compared with searching enormous directory structures.
On the small-screen web, including mobile phones, the costs of ordinary interaction are relatively higher and so, as pointed out by one of the authors in a keynote even back in 1999, the advantages of using ‘intelligent’ techniques are comparatively greater [22]. Similar considerations are driving HP Labs’ “Simplifying Web Access for the Next billion” (SWAN) project.2
In this paper we discuss several technologies offering the user automated task support, and in particular weaving together fragments of web and desktop interaction, so that the coherence in the user's mind is to some extent also reflected in the system. To be effective, such interactions have to fit with the human user; we are looking for ‘appropriate intelligence’, a blend of user controlled and computer aided activity set within a context of interaction that is meaningful and beneficial to the user.
Some of this currently uses standard semantic technology, some more bespoke representations. We will describe how the shared semantic representations promised by the Semantic Web can aid this integration, and discuss some of the obstacles and hence challenges for the development of core Semantic Web technology.
The decomposition of software enabled by mash-ups, plug-ins and widgets has tremendous potential for the democratisation of software, offering an alternative to behemoth applications and the stranglehold of massive vendors. To attain this, however, we need better ways for micro-applications to work together rather than just plug into larger software; a sit-alongside model for future software.
The next section will elaborate the motivation and background for this work, presenting a motivating scenario, a description of human activity that is both the context and also inspiration for the automated support we provide, and a short review of other ways global data is used to help user interaction. Section 3 then goes on to present the four core components of our wider vision for task support, reviewing additional literature and related systems for each as appropriate. In Section 4, issues of integration are discussed, based largely on two prototype systems, a web-based application Snip!t [27], [30] and a desktop system On Time [13], which bring together various of the components discussed in Section 3. This integration experience is analysed in terms of architecture, inter-operability and user interaction. Finally Section 5 looks at implications and issues for the Semantic Web and web-based user activity arising from the experiences of design, development and deployment outlined in the previous sections.
This paper is partly about our own existing work, dating back over 10 years and its ongoing trajectory [23], [26], [27], [31]. However, it is also about a vision common with others such as the HP SWAN project, Heath et al.’s call for a task-focused web [43], and Berners-Lee's ‘underground map’ of the future web landscape [7]; a vision of interactions formed through small units of task-based web activity being linked together by and for users to create a more seamless next generation web experience.
So, while the authors’ own components, technologies and systems are described in some detail, the intention is not so much to promote these aspects of our own work, but more to use them as a proof of concept of this wider vision; using our own experiences to populate an initial roadmap for the future.
Section snippets
Origins of the work
We have come to this work through a number of roots, but brought together in the TIM (Task-centred Information Management) project [56], [9], part of the DELOS EU Network of Excellence on Digital Libraries. The backgrounds of the team included formal ontologies, ontology visualization, databases and intelligent internet user interfaces. Research within the field of Personal Information Management (PIM) [52], as its name suggests, is focused mostly on the user's information resources: calendars,
Components for user task support
In this section we present the core technologies we are using or developing in order to attain automated task assistance. First is the use of a personal ontology as a repository and spreading activation in order to model memory and context. Second is data detector technology, which can be used to turn unstructured data into the locus for interaction, thus triggering activity. Third are algorithms to help users during specific actions by inferring relationships between form fields, linking data
Putting it together
Having seen the different component technologies, we now discuss how they fit together architecturally and in terms of user interaction.
Challenges for a web of action
Each core technology has its own challenges, as does the integration process. Some of these are made simpler by a more pragmatic approach (e.g. making multiple suggestions to the user in cases of uncertainty), but others are made more complex. In this section we will consider some of the issues that are highlighted by work in the area to date, and those that are likely to become important in the near future.
Summary and ongoing work
This paper has presented technology targeted at producing a web of action, where our day-to-day activities not only have information at hand, but also are actively supported as activity, not merely information seeking. We have presented a number of technologies, which address different aspects of automated task support based on the categorisation of human activity in Table 1.
The personal ontology is used to enable the system to share some of the user's knowledge of the world, assisted by
Acknowledgements
Parts of this work were supported by the Information Society Technologies (IST) Program of the European Commission as part of the DELOS Network of Excellence on Digital Libraries (Contract G038-507618). Thanks also to Emanuele Tracanna, Marco Piva, and Raffaele Giuliano for their work on On Time.
References (76)
A spreading activation theory of memory
Journal of Verbal Learning and Verbal Behaviour
(1983)- et al.
Magpie: experiences in supporting semantic web browsing
Web Semantics: Science, Services and Agents on the World Wide Web
(2007) - et al.
Ontologies and the brain: using spreading activation through ontologies to support personal interaction
Cognitive Systems Research
(2010) - C.R. Atkinson, M.R. Shiffrin, Human memory: a proposed system and its control processes, in: K.W. Spence, J.T. Spence...
- N. Ashish, C.A. Knoblock, Semi-automatic wrapper generation for internet information sources, in: Proceedings of the...
- BBC Backstage, British Broadcasting Corporation, dated 2004–2005, Accessed on 6/7/2009,...
- C. Bizer, T. Heath, T. Berners-Lee, Linked Data. The Story So Far, International Journal on Semantic Web and...
- T. Berners-Lee, Looking Back, Looking Forward: The Process of Designing Things in a Very Large Space, Inaugural...
- S. Brin, L. Page, The anatomy of a large-scale hypertextual web search engine, in: Proceedings of Seventh ACM...
Tractable reasoning and efficient query answering in description logics: the DL-Lite family
Journal of Automated Reasoning
Being there: putting brain
Body and the World Together Again
A spreading-activation theory of semantic processing
Psychological Review
Application of spreading activation techniques in information retrieval
Artificial Intelligence Review
A conceptual framework and a toolkit for supporting the rapid prototyping of context-aware applications
Human–Computer Interaction
The Handbook of Task Analysis for Human–Computer Interaction
Human–Computer Interaction
Toward a theory of situation awareness in dynamic systems
Human Factors
Long-term working memory
Psychological Review
Cited by (12)
“Every Website Is a Puzzle!”: Facilitating Access to Common Website Features for People with Visual Impairments
2022, ACM Transactions on Accessible ComputingTruth in an Age of Information
2022, International Conference on Computer-Human Interaction Research and Applications, CHIRA - ProceedingsLearning to represent human motives for goal-directed web browsing
2021, RecSys 2021 - 15th ACM Conference on Recommender SystemsA Web-Based Decision Support System for Supply Chain Operations Management Towards an Integrated Framework
2014, Lecture Notes in Business Information ProcessingThe influence of proactivity on interactive help agents
2013, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)