Elsevier

Journal of Web Semantics

Volume 9, Issue 4, December 2011, Pages 474-489
Journal of Web Semantics

Relevance feedback between hypertext and Semantic Web search: Frameworks and evaluation

https://doi.org/10.1016/j.websem.2011.10.001Get rights and content

Abstract

We investigate the possibility of using Semantic Web data to improve hypertext Web search. In particular, we use relevance feedback to create a ‘virtuous cycle’ between data gathered from the Semantic Web of Linked Data and web-pages gathered from the hypertext Web. Previous approaches have generally considered the searching over the Semantic Web and hypertext Web to be entirely disparate, indexing, and searching over different domains. While relevance feedback has traditionally improved information retrieval performance, relevance feedback is normally used to improve rankings over a single data-set. Our novel approach is to use relevance feedback from hypertext Web results to improve Semantic Web search, and results from the Semantic Web to improve the retrieval of hypertext Web data. In both cases, an evaluation is performed based on certain kinds of informational queries (abstract concepts, people, and places) selected from a real-life query log and checked by human judges. We evaluate our work over a wide range of algorithms and options, and show it improves baseline performance on these queries for deployed systems as well, such as the Semantic Web Search engine FALCON-S and Yahoo! Web search. We further show that the use of Semantic Web inference seems to hurt performance, while the pseudo-relevance feedback increases performance in both cases, although not as much as actual relevance feedback. Lastly, our evaluation is the first rigorous ‘Cranfield’ evaluation of Semantic Web search.

Introduction

There has recently been a return of interest in semantic search. This seems inspired in part by the Semantic Web, in particular by the Linked Data initiative’s releasing of a massive amount of public structured data on the Web from a diverse range of sources in common Semantic Web formats like RDF (Resource Description Format) [16]. This has in turn led to the rise of specialized Semantic Web search engines. Semantic Web search engines are search engines that specifically index and return ranked Linked Data in RDF in response to keyword queries, but their rankings are much less well-studied than hypertext Web rankings, and so are thought likely to be sub-optimal.

One hypothesis put forward by Baeza-Yates is that the search on the Semantic Web can be used to improve traditional ad-hoc information retrieval for hypertext Web search engines and vice-versa [2]. While we realize the amount and sources of structured data on the Web are huge, to restrict and test the hypothesis of Baeza-Yates, from hereon we will assume that ‘semantic search’ refers to indexing and retrieving of Linked Data by search engines like Sindice and FALCON-S [7], and hypertext search refers to the indexing and retrieval of hypertext documents on the World Wide Web by search engines like Google and Yahoo! Search.

We realize that our reduction of ‘semantic search’ to keyword-based information retrieval over the Semantic Web is very restrictive, as many people use ‘semantic search’ to mean simply search that relies on anything beyond surface syntax, including the categorization of complex queries [3] and entity-recognition using Semantic Web ontologies [12]. We will not delve into an extended explanation of the diverse kinds of semantic search, as surveys of this kind already exist [20]. Yet given the relative paucity of publicly accessible data-sets about the wider notion of semantics and the need to start with a simple rather than complex paradigm, we will restrict ourselves to the Semantic Web and assume a traditional, keyword-based ad-hoc information retrieval paradigm for both kinds of search, leaving issues like complex queries and natural language semantics for future research. Keyword search consisting of 1–2 terms should also be explored as it is the most common kind of query in today’s Web search regardless of whether any results from this experiment can generalize to other kinds of semantic search [27].

Until recently semantic search suffered from a lack of a thorough and neutral Cranfield-style evaluation, and so we carefully explain and employ the traditional information retrieval evaluation frameworks in our experiment to evaluate semantic search. At the time of the experiment, our evaluation was the first Cranfield-style evaluation for searching on the Semantic Web. This evaluation later generalized into the annual ‘semantic search’ competition,2 which has since become a standard evaluation for search over RDF data [4]. However, our particular evaluation presented here is still the only evaluation to determine relevance judgments over both hypertext and RDF using the same set of queries.

Section snippets

Overview

Our hypothesis is that relevance feedback can improve the ranking of relevant results for both hypertext Web search and Semantic Web search. Previous approaches have assumed that the Semantic Web and the hypertext Web to be entirely disparate, indexing and searching them differently [7]. Our novel approach is to use relevance feedback from hypertext Web search to improve the retrieval of semantic search. Then more interestingly, we attempt to run the relevance feedback in reverse: Assuming we

Is there anything worth finding on the Semantic Web?

In this section we demonstrate that the Semantic Web does indeed contain information relevant to ordinary users by sampling the Semantic Web according to a real-world queries referring to entities and concepts from the query log of a major search engine. The main problem confronting of any study of the Semantic Web is one of sampling. As almost any large-data database can easily be exported to RDF, statistics demonstrating the actual deployment of the Semantic Web can be biased by the automated

Information retrieval for web search

In our evaluation we tested two general kinds of information retrieval frameworks: vector-space models and language models. In the vector-space model, document models are considered to be vectors of terms (usually called ‘words’ as they are usually, although not exclusively, from natural language, as we transform URIs into ‘pseudo-words’) where the weighing function and query expansion has no principled basis besides empirical results. Ranking is usually done via a comparison using the cosine

System description

We present a novel system that uses the same underlying information retrieval system on both hypertext and Semantic Web data so that relevance feedback can be done in a principled manner from both sources of data with language models. In our system, the query is run first against the hypertext Web and relevant hypertext results can then be used to expand a Semantic Web search query with terms from resulting hypertext web-pages. The expanded query is then ran against the Semantic Web, resulting

Feedback evaluation

In this section we evaluate algorithms and parameters using relevance feedback against the same system without relevance feedback. In Section 9 we evaluate against deployed systems such as FALCON-S and Yahoo! Web Search. To preview our final results in Section 9, relevance feedback from the Semantic Web shows an impressive 25% gain in average precision over Yahoo! Web Search with a 16% gain in precision over FALCON-S without relevance feedback.

Pseudo-feedback

In this section we explore a very easy-to-implement and feasible way to take advantage of relevance feedback without manual selection of relevant results by human users. One of the major problems of relevance feedback-based approaches is their dependence on manual selection of relevant results by human users. For example, in our experiments we used judges manually determining if web-pages were relevant using an experimental set-up that forced them to judge every result as relevant or not, which

Inference

In this section the effect of inference on relevance feedback is evaluated by considering inference to be document expansion. One of the characteristics of the Semantic Web is that the structure should allow one ‘in theory’ to discover more relevant data. The Semantic Web formalizes this in terms of type and sub-class hierarchies in RDF using RDF Schema [6]. While inference routines are quite complicated as regards the various Semantic Web specifications, in practice the vast majority of

Deployed systems

In this section we evaluate our system against ‘real-world’ deployed systems. One area we have not explored is how systems based on relevance feedback perform relative to systems that are actually deployed, as our previous work has always been evaluated against systems and parameters we created specifically for experimental evaluation. Our performance in Section 6.1.1 and Section 6.2.1 was only compared to baselines that were versions of our weighting function without a relevance feedback

Future work

There are a number of areas where our project needs to be more thoroughly integrated with other approaches and improved. The expected criticism of this work is likely the choice of FALCON-S and Yahoo! Web search as a baseline, and that we should try this methodology over other Semantic Web search engines and hypertext Web search engines. Lastly, currently it is unknown how to combine traditional word-based techniques from information retrieval with structural techniques from the Semantic Web,

Conclusion

This study features a number of results that impact the larger field of semantic search. First, it shows a rigorous information retrieval evaluation, the ‘Cranfield paradigm’, can be applied to semantic search despite the differences between the Semantic Web and hypertext. These differences are well-recorded in our sample of the Semantic Web as taken via FALCON-S using a query log, and reveals a number of large differences between the Semantic Web data and hypertext data, in particular that

References (28)

  • J. Allan, M. Connell, W.B. Croft, F.F. Feng, D. Fisher, X. Li, INQUERY and TREC-9, in: Proceedings of the Ninth Text...
  • R. Baeza-Yates, From capturing semantics to semantic search: a virtuous cycle, in: Proceedings of the 5th European...
  • R.A. Baeza-Yates, A. Tiberi, Extracting semantic relations from query logs, in: Proceedings of the Conference on...
  • R. Blanco et al.

    Repeatable and reliable search system evaluation using crowdsourcing

  • R. Blanco et al.

    Entity search evaluation over structured web data

  • D. Brickley, R.V. Guha, RDF Vocabulary Description Language 1.0: RDF Schema, Recommendation, W3C, 2004,...
  • G. Cheng, W. Ge, Y. Qu, FALCONS: Searching and browsing entities on the Semantic Web, in: Proceedings of the World Wide...
  • N. Craswell, H. Zaragoza, S. Robertson, Microsoft Cambridge at trec-14: Enterprise track, in: Proceedings of the...
  • H. Cui et al.

    Probabilistic query expansion using query logs

  • L. Ding, T. Finin, Characterizing the semantic web on the web, in: Proceedings of the International Semantic Web...
  • J. Fleiss

    Measuring nominal scale agreement among many raters

    Psychological Bulletin

    (1971)
  • R. Guha et al.

    Semantic search

  • H. Halpin, A query-driven characterization of linked data, in: Proceedings of the Linked Data Workshop at the World...
  • D. Hawking et al.

    Overview of the trec-8 web track

  • Cited by (10)

    • Supporting inter-topic entity search for biomedical Linked Data based on heterogeneous relationships

      2017, Computers in Biology and Medicine
      Citation Excerpt :

      In order to provide better results, two methods are used in general for: (1) query expansions, and (2) filters in advanced search. Query expansion tends to retrieve more research results by enriching the keyword query with expansions, which are the related term(s) (e.g., synonym, hypernym) based on knowledge-bases [11] or Web of document [12], statistical related entities based on co-occurrence [13], or the preferences [14]. Filters use additional input, e.g., search preferences, to narrow down the search space to specific topics to retrieve more precise results.

    • Knowledge-based personalized search engine for the Web-based Human Musculoskeletal System Resources (HMSR) in biomechanics

      2013, Journal of Biomedical Informatics
      Citation Excerpt :

      Consequently, these conceptual and technological formalisms show potential perspectives to develop our knowledge-based personalized search engine in the Biomechanics field. A significant number of semantic search engines have been also developed [28–33]. New keywords-based [29,34] or map-based [30] or graph-based [35] search strategies have been developed recently to provide user-friendly query approaches as well as to improve the accuracy of the retrieved results.

    • Source selection of long tail sources for federated search in an uncooperative setting

      2018, Proceedings of the ACM Symposium on Applied Computing
    • Digital Libraries: Interoperability and Uses

      2016, Digital Libraries: Interoperability and Uses
    • Digital Libraries

      2016, Digital Libraries
    View all citing articles on Scopus
    1

    Support provided by Microsoft “Beyond Search” award.

    View full text