Relevance feedback between hypertext and Semantic Web search: Frameworks and evaluation
Introduction
There has recently been a return of interest in semantic search. This seems inspired in part by the Semantic Web, in particular by the Linked Data initiative’s releasing of a massive amount of public structured data on the Web from a diverse range of sources in common Semantic Web formats like RDF (Resource Description Format) [16]. This has in turn led to the rise of specialized Semantic Web search engines. Semantic Web search engines are search engines that specifically index and return ranked Linked Data in RDF in response to keyword queries, but their rankings are much less well-studied than hypertext Web rankings, and so are thought likely to be sub-optimal.
One hypothesis put forward by Baeza-Yates is that the search on the Semantic Web can be used to improve traditional ad-hoc information retrieval for hypertext Web search engines and vice-versa [2]. While we realize the amount and sources of structured data on the Web are huge, to restrict and test the hypothesis of Baeza-Yates, from hereon we will assume that ‘semantic search’ refers to indexing and retrieving of Linked Data by search engines like Sindice and FALCON-S [7], and hypertext search refers to the indexing and retrieval of hypertext documents on the World Wide Web by search engines like Google and Yahoo! Search.
We realize that our reduction of ‘semantic search’ to keyword-based information retrieval over the Semantic Web is very restrictive, as many people use ‘semantic search’ to mean simply search that relies on anything beyond surface syntax, including the categorization of complex queries [3] and entity-recognition using Semantic Web ontologies [12]. We will not delve into an extended explanation of the diverse kinds of semantic search, as surveys of this kind already exist [20]. Yet given the relative paucity of publicly accessible data-sets about the wider notion of semantics and the need to start with a simple rather than complex paradigm, we will restrict ourselves to the Semantic Web and assume a traditional, keyword-based ad-hoc information retrieval paradigm for both kinds of search, leaving issues like complex queries and natural language semantics for future research. Keyword search consisting of 1–2 terms should also be explored as it is the most common kind of query in today’s Web search regardless of whether any results from this experiment can generalize to other kinds of semantic search [27].
Until recently semantic search suffered from a lack of a thorough and neutral Cranfield-style evaluation, and so we carefully explain and employ the traditional information retrieval evaluation frameworks in our experiment to evaluate semantic search. At the time of the experiment, our evaluation was the first Cranfield-style evaluation for searching on the Semantic Web. This evaluation later generalized into the annual ‘semantic search’ competition,2 which has since become a standard evaluation for search over RDF data [4]. However, our particular evaluation presented here is still the only evaluation to determine relevance judgments over both hypertext and RDF using the same set of queries.
Section snippets
Overview
Our hypothesis is that relevance feedback can improve the ranking of relevant results for both hypertext Web search and Semantic Web search. Previous approaches have assumed that the Semantic Web and the hypertext Web to be entirely disparate, indexing and searching them differently [7]. Our novel approach is to use relevance feedback from hypertext Web search to improve the retrieval of semantic search. Then more interestingly, we attempt to run the relevance feedback in reverse: Assuming we
Is there anything worth finding on the Semantic Web?
In this section we demonstrate that the Semantic Web does indeed contain information relevant to ordinary users by sampling the Semantic Web according to a real-world queries referring to entities and concepts from the query log of a major search engine. The main problem confronting of any study of the Semantic Web is one of sampling. As almost any large-data database can easily be exported to RDF, statistics demonstrating the actual deployment of the Semantic Web can be biased by the automated
Information retrieval for web search
In our evaluation we tested two general kinds of information retrieval frameworks: vector-space models and language models. In the vector-space model, document models are considered to be vectors of terms (usually called ‘words’ as they are usually, although not exclusively, from natural language, as we transform URIs into ‘pseudo-words’) where the weighing function and query expansion has no principled basis besides empirical results. Ranking is usually done via a comparison using the cosine
System description
We present a novel system that uses the same underlying information retrieval system on both hypertext and Semantic Web data so that relevance feedback can be done in a principled manner from both sources of data with language models. In our system, the query is run first against the hypertext Web and relevant hypertext results can then be used to expand a Semantic Web search query with terms from resulting hypertext web-pages. The expanded query is then ran against the Semantic Web, resulting
Feedback evaluation
In this section we evaluate algorithms and parameters using relevance feedback against the same system without relevance feedback. In Section 9 we evaluate against deployed systems such as FALCON-S and Yahoo! Web Search. To preview our final results in Section 9, relevance feedback from the Semantic Web shows an impressive 25% gain in average precision over Yahoo! Web Search with a 16% gain in precision over FALCON-S without relevance feedback.
Pseudo-feedback
In this section we explore a very easy-to-implement and feasible way to take advantage of relevance feedback without manual selection of relevant results by human users. One of the major problems of relevance feedback-based approaches is their dependence on manual selection of relevant results by human users. For example, in our experiments we used judges manually determining if web-pages were relevant using an experimental set-up that forced them to judge every result as relevant or not, which
Inference
In this section the effect of inference on relevance feedback is evaluated by considering inference to be document expansion. One of the characteristics of the Semantic Web is that the structure should allow one ‘in theory’ to discover more relevant data. The Semantic Web formalizes this in terms of type and sub-class hierarchies in RDF using RDF Schema [6]. While inference routines are quite complicated as regards the various Semantic Web specifications, in practice the vast majority of
Deployed systems
In this section we evaluate our system against ‘real-world’ deployed systems. One area we have not explored is how systems based on relevance feedback perform relative to systems that are actually deployed, as our previous work has always been evaluated against systems and parameters we created specifically for experimental evaluation. Our performance in Section 6.1.1 and Section 6.2.1 was only compared to baselines that were versions of our weighting function without a relevance feedback
Future work
There are a number of areas where our project needs to be more thoroughly integrated with other approaches and improved. The expected criticism of this work is likely the choice of FALCON-S and Yahoo! Web search as a baseline, and that we should try this methodology over other Semantic Web search engines and hypertext Web search engines. Lastly, currently it is unknown how to combine traditional word-based techniques from information retrieval with structural techniques from the Semantic Web,
Conclusion
This study features a number of results that impact the larger field of semantic search. First, it shows a rigorous information retrieval evaluation, the ‘Cranfield paradigm’, can be applied to semantic search despite the differences between the Semantic Web and hypertext. These differences are well-recorded in our sample of the Semantic Web as taken via FALCON-S using a query log, and reveals a number of large differences between the Semantic Web data and hypertext data, in particular that
References (28)
- J. Allan, M. Connell, W.B. Croft, F.F. Feng, D. Fisher, X. Li, INQUERY and TREC-9, in: Proceedings of the Ninth Text...
- R. Baeza-Yates, From capturing semantics to semantic search: a virtuous cycle, in: Proceedings of the 5th European...
- R.A. Baeza-Yates, A. Tiberi, Extracting semantic relations from query logs, in: Proceedings of the Conference on...
- et al.
Repeatable and reliable search system evaluation using crowdsourcing
- et al.
Entity search evaluation over structured web data
- D. Brickley, R.V. Guha, RDF Vocabulary Description Language 1.0: RDF Schema, Recommendation, W3C, 2004,...
- G. Cheng, W. Ge, Y. Qu, FALCONS: Searching and browsing entities on the Semantic Web, in: Proceedings of the World Wide...
- N. Craswell, H. Zaragoza, S. Robertson, Microsoft Cambridge at trec-14: Enterprise track, in: Proceedings of the...
- et al.
Probabilistic query expansion using query logs
- L. Ding, T. Finin, Characterizing the semantic web on the web, in: Proceedings of the International Semantic Web...
Measuring nominal scale agreement among many raters
Psychological Bulletin
Semantic search
Overview of the trec-8 web track
Cited by (10)
Supporting inter-topic entity search for biomedical Linked Data based on heterogeneous relationships
2017, Computers in Biology and MedicineCitation Excerpt :In order to provide better results, two methods are used in general for: (1) query expansions, and (2) filters in advanced search. Query expansion tends to retrieve more research results by enriching the keyword query with expansions, which are the related term(s) (e.g., synonym, hypernym) based on knowledge-bases [11] or Web of document [12], statistical related entities based on co-occurrence [13], or the preferences [14]. Filters use additional input, e.g., search preferences, to narrow down the search space to specific topics to retrieve more precise results.
Knowledge-based personalized search engine for the Web-based Human Musculoskeletal System Resources (HMSR) in biomechanics
2013, Journal of Biomedical InformaticsCitation Excerpt :Consequently, these conceptual and technological formalisms show potential perspectives to develop our knowledge-based personalized search engine in the Biomechanics field. A significant number of semantic search engines have been also developed [28–33]. New keywords-based [29,34] or map-based [30] or graph-based [35] search strategies have been developed recently to provide user-friendly query approaches as well as to improve the accuracy of the retrieved results.
Source selection of long tail sources for federated search in an uncooperative setting
2018, Proceedings of the ACM Symposium on Applied ComputingDigital Libraries: Interoperability and Uses
2016, Digital Libraries: Interoperability and UsesDigital Libraries
2016, Digital LibrariesDiscovering expansion entities for keyword-based entity search in linked data
2015, Journal of Information Science
- 1
Support provided by Microsoft “Beyond Search” award.