From the web of data to a world of action

Abstract

This paper takes as its premise that the web is a place of action, not just information, and that the purpose of global data is to serve human needs. The paper presents several component technologies, which together work towards a vision where many small micro-applications can be threaded together using automated assistance to enable a unified and rich interaction. These technologies include data detector technology to enable any text to become a start point of semantic interaction; annotations for web-based services so that they can link data to potential actions; spreading activation over personal ontologies, to allow modelling of context; algorithms for automatically inferring ‘typing’ of web-form input data based on previous user inputs; and early work on inferring task structures from action traces. Some of these have already been integrated within an experimental web-based (extended) bookmarking tool, Snip!t, and a prototype desktop application On Time, and the paper discusses how the components could be more fully, yet more openly, linked in terms of both architecture and interaction. As well as contributing to the goal of an action and activity-focused web, the work also exposes a number of broader issues, theoretical, practical, social and economic, for the Semantic Web.

Introduction

From human readable web pages, to formal semantics of linked data and the emergent social semantics of tags and folksonomies, we routinely look to the web as both a source of information and a place to put data. However, the web is also a locus of action: users want to get things done, whether booking a hotel room, or editing an online spreadsheet.

The boundaries between web and desktop interaction are blurring. On the one hand, the traditional PC desktop is now inhabited by widgets such as the Mac Dashboard, web fast-download apps such as Java Web Start or Adobe Air, and expanded browser functionality such as Chrome. On the other hand, computation and applications that once were part of the desktop are now hosted on the web (for example word-processing with Google Docs), and various technologies enable web applications to function even when users have no connectivity to the internet (for example DojoX [33], Google Gears and the offline mode of HTML5 [75], [76]). Furthermore, in emerging markets such as India and China, this convergence will be total, as the sole computing experience for many will be through mobile devices and predominantly the web.

So far most of these web-like or web-based applications are separate, and web activity is glued together by the user, often through crude cutting and pasting between web applications. This separation is epitomised by Google Web Elements,¹ which embed content in users’ web pages, but, with the exception of Google maps, are largely sealed from one another and their context; while considerably more functional than data feeds, in the end no more integrated than early web syndication. Embedded applications, such as those in Facebook or Google Widgets, are integrated more richly with their respective underlying platforms, but again largely firewalled from each other, preventing synergistic interactions.

However, there is an emerging need to offer greater support to users in performing web-based activity that cut across individual applications, and potentially to partially automate common tasks.

For over 20 years the dominant interface paradigm has been instrumental: populating the interface with virtual ‘things’ (documents, shapes, files as icons) that are made as transparent as possible and manipulated ‘directly’ by the user [67], [46]. However, the balance is changing and a level of ‘intelligent’, mediated interaction is becoming more accepted. This is partly because Moore's law means that it is easier to do more clever things, but more significantly because of the changing environment.

On the big-screen web (web on a desktop or laptop PC), this is largely due to the sheer size of data available on the web, so that Google search or Amazon suggestions become acceptable compared with searching enormous directory structures.

On the small-screen web, including mobile phones, the costs of ordinary interaction are relatively higher and so, as pointed out by one of the authors in a keynote even back in 1999, the advantages of using ‘intelligent’ techniques are comparatively greater [22]. Similar considerations are driving HP Labs’ “Simplifying Web Access for the Next billion” (SWAN) project.²

In this paper we discuss several technologies offering the user automated task support, and in particular weaving together fragments of web and desktop interaction, so that the coherence in the user's mind is to some extent also reflected in the system. To be effective, such interactions have to fit with the human user; we are looking for ‘appropriate intelligence’, a blend of user controlled and computer aided activity set within a context of interaction that is meaningful and beneficial to the user.

Some of this currently uses standard semantic technology, some more bespoke representations. We will describe how the shared semantic representations promised by the Semantic Web can aid this integration, and discuss some of the obstacles and hence challenges for the development of core Semantic Web technology.

The decomposition of software enabled by mash-ups, plug-ins and widgets has tremendous potential for the democratisation of software, offering an alternative to behemoth applications and the stranglehold of massive vendors. To attain this, however, we need better ways for micro-applications to work together rather than just plug into larger software; a sit-alongside model for future software.

The next section will elaborate the motivation and background for this work, presenting a motivating scenario, a description of human activity that is both the context and also inspiration for the automated support we provide, and a short review of other ways global data is used to help user interaction. Section 3 then goes on to present the four core components of our wider vision for task support, reviewing additional literature and related systems for each as appropriate. In Section 4, issues of integration are discussed, based largely on two prototype systems, a web-based application Snip!t [27], [30] and a desktop system On Time [13], which bring together various of the components discussed in Section 3. This integration experience is analysed in terms of architecture, inter-operability and user interaction. Finally Section 5 looks at implications and issues for the Semantic Web and web-based user activity arising from the experiences of design, development and deployment outlined in the previous sections.

This paper is partly about our own existing work, dating back over 10 years and its ongoing trajectory [23], [26], [27], [31]. However, it is also about a vision common with others such as the HP SWAN project, Heath et al.’s call for a task-focused web [43], and Berners-Lee's ‘underground map’ of the future web landscape [7]; a vision of interactions formed through small units of task-based web activity being linked together by and for users to create a more seamless next generation web experience.

So, while the authors’ own components, technologies and systems are described in some detail, the intention is not so much to promote these aspects of our own work, but more to use them as a proof of concept of this wider vision; using our own experiences to populate an initial roadmap for the future.