GrandBase: generating actionable knowledge from Big Data

3.1.1 Predicate extraction from existing KBs.

We use two dominate existing KBs, namely, Freebase (covers 4, 000 predicates) and DBpedia (covers 6, 000 predicates). We first analyze each KB separately by applying the intuition that each sub-type should inherit all the predicates of its super-types. Then we combine the predicate extractions from both KBs. In particular, we attach the predicates of each DBpedia class to the similar[5] type in Freebase. We also apply duplication removal for the combined predicates to avoid redundancy, by comparing the predicates of similar class or type in both KBs in terms of name, label and comment/description. For example, for the type “Book”, there are 5 predicates attached to it in Freebase and 21 in DBpedia. After inheriting all the predicates from the corresponding super-types of “Book” in the corresponding KB, we can get 19 predicates extracted from Freebase and 48 from DBpedia. By combining those predicate extractions and applying duplication removal, we finally obtain 60 predicates to be attached to Freebase.

3.1.2 Predicate extraction from query stream.

As query stream naturally captures users’ collective convictions on possible predicates of entities, it is a high-quality resource for predicate extraction. We define the query stream extractor by using more patterns than previous methods, including “what/how/when/who is the p of (the/a/an) e”, “the p of (the/a/an) e” and “e’s p” (where e represents an entity and p depicts a possible predicate), and a set of filtering rules. Our designed patterns are applied to extract more possible predicates, while the filtering rules are used to purify the extractions. Specifically, our query stream extractor conducts a five-step process to extract predicates for each type (denoted as T) in Freebase.

The initial step is relevant query stream identification. We apply an entity recognizer to identify the queries that contain entities belonging to T, and denote the set of relevant queries as Q_R(T). Second, we conduct predicate candidate identification. From Q_R(T), our extractor finds all the queries that match any of our predefined patterns. We denote those queries as Q_P(T) and add the identified predicate candidates to the set P(T). For example, given a relevant query “Taken 3’s box office” of type movie, as it matches the pattern “e’s p”, we add “box office” to P(movie) as a predicate candidate of movie and this query to Q_P(movie). Note that P(T) may be noisy and contain non-predicate elements. For example, both queries “The University of Adelaide” and “the plural of country” match our predefined pattern “the p of (the/a/an) e”, but “University” is not actually a predicate of “Adelaide”, so as is “plural”. Therefore, our third step focuses on filtering out non-predicates in P(T). To this end, we sample a set of relevant queries that are related to various types, and extract predicate candidates for each type. Then, we rank the predicate candidates by the number of types they belong to, and add the top predicate candidates to a blacklist to avoid the appearance of the non-predicate elements, such as “lack”, “rest”, “meaning”, “best”, “definition”, “summary” and “plural”, in P(T). For such cases as “The University of Adelaide”, we apply named entity identification to remove this type of non-predicates from P(T). In particular, we compare each query in Q_P(T) with the entities of T in Freebase, and if they match each other, we remove the corresponding predicate from P(T). In the fourth step, we conduct entity–predicate pair identification. For each query in Q_R(T), if the query containing both e belongs to T and p ∈ P(T), we add the corresponding (p, e) pair to a set PE. To further purify the predicate extractions, we conduct the fifth step, credible predicate identification, by applying the following two rules. The result set P(T) is attached to the type T in Freebase for ontology augmentation.

Rule 1: Given a type T, which contains N entities {e₁, e₂, …, e_N}, and a predicate p, p will not be attached to T, if ∃e_j ∈ {e₁, e₂, …, e_N}, EntityFrequency(e_j) ≥ max(p,e*)∈PE (EntityFrequency(e*)) and (p, e_j) ∉ PE.

Here EntityFrequency(e) represents the number of queries in Q_R(T) that cover a specific entity e.

Rule 2: For each EntityDiversity(T, p) ≠ 0, we remove p from P(T), if EntityDiversity(T,p)N≤α (a pre-defined threshold).

Here EntityDiversity(T, p) depicts the number of distinct entities of T which co-appear with a predicate p in PE.

3.1.3 Predicate extraction from DOM trees.

Typically, Web pages are semi-structured and described by nested HTML tags. The tree-like structures can be commonly found in Web pages that contain Web lists and Web tables, as well as deep-Web sources, and are regarded as DOM trees[6]. To explore knowledge from Web pages, traditional extractors simply remove the HTML tags and process the plain texts. Thus, they fail to exploit the knowledge contained in the DOM trees. We propose a two-step extractor to extract predicates from DOM trees to augment the ontology of Freebase: first, seeded by the predicate extractions from existing KBs and query stream, we extract additional predicates from the DOM trees; second, we purify the extractions and differentiate the quality of the extracted predicates by applying a set of filters.

At the first step, our approach alternatively extracts predicates and learns tag path patterns through an iterative process. The detailed procedure is described in Algorithm 1. Briefly, given a type T in Freebase, the algorithm first identifies the relevant Websites of T (e.g. www.imdb.com/for type Film), denoted by S(T). For each Web page wp_s ∈ s and s ∈ S(T), the algorithm analyzes the DOM structure and classifies the text nodes into entity nodes (i.e. the texts represent an entity e that belongs to T) and non-entity nodes. The tag paths between each entity node and its corresponding non-entity node are then extracted. After the removal of noisy tags, the extracted tag paths are kept in a tag path set, denoted by TP(wp_s) (Line 1). For each website s ∈ S(T) (Line 2), the algorithm iteratively finds the Web pages that contain at least one (p, e) pair, where e belongs to T, p ∈ seed(T) and seed(T) is the seed predicates extracted from existing KBs and query stream. For each eligible Web page wp_s^* (Line 3), the algorithm traverses TP(wp_s^*) for this Web page to obtain the tag paths between the seed p and the corresponding e, and transfers these tag paths from TP(wp_s^*) to an induced tag path pattern set, denoted by TP_I(wp_s^*) (Line 4). We next compare all the tag paths in the TP(wp_s^*) with the patterns in TP_I(wp_s^*) (Line 5). Those non-entity nodes with similar tag paths with the induced patterns are finally recognized as predicates (Line 6). If a predicate is not covered by seed(T), then it is added to seed(T) for augmentation (Line 7), with the corresponding tag paths removed from TP(wp_s^*) (Line 8). The algorithm turns to another Website when the number of predicates in seed(T) reaches a certain threshold. Because the number of Web pages and text nodes in a Web page is limited, the algorithm can always converge.

As the raw predicates extracted by the first step may contain considerable noises, in the second step, we use the following three types of features to purify the extracted predicates:

1. The inherent features of a predicate displayed in a Web page. A node that denotes a predicate in a DOM tree always follows some inherent rules, e.g. the length of the text is always limited to a certain number of words (we discover that almost all predicates in Freebase is described by less than ten words), the text node always contains a colon as the end of the string or the first letter of every word in the text node is in upper case (Web page always capitalizes the first letter for each word of the name of a predicate).

2. The intra-site features of a predicate displayed in a Web page. If a predicate is described by a Website, it tends to appear frequently in a considerable number of pages of this website.

3. The inter-site features of a predicate displayed in a Web page.

Predicates tend to appear in multiple websites instead of very few websites.

We can simply neglect the extractions that mismatch these features, but this may result in some loss of recall. For example, in reality, the number of movies that win an Oscar award is quite limited. Thus, the predicate “winner in Oscar” would not appear frequently in the Web pages of a movie website, dissatisfying the second feature. If we strictly use the feature, we tend to not regard “winner in Oscar” as a predicate by mistake. Therefore, we sequentially use three filters to deliver three predicate sets in turn, i.e. potential predicates, predicate candidates and credible predicates. Each set represents a different balance between the precision and recall of the extractions and can be fed to knowledge-driven applications based on their own requirements.

In particular, we apply the inherent feature filter to obtain the set of potential predicates by using the specific rules followed in the DOM trees. By leveraging the intra-site feature filter, we remove all predicates with intra-site frequency lower than a predefined threshold β to obtain the predicate candidate set. We calculate the intra-site frequency of a predicate p in a website s by the following equation:

The intra-site feature filter has limitations, as it may incorrectly take some Website-specific terms as predicates. For example, for the sake of display, a node with text “edit” appears frequently in IMDb (a famous movie Website). In this case, the intra-site feature filter would incorrectly identify edit as a predicate of type Movie. However, we notice that such predicates are often website-specific, which is in contrast seldom contained by other Web sites. Therefore, we can remove such noises by additionally examining the inter-site frequency feature of the predicate extractions. To construct the inter-site feature filter, we should not only consider the sum of the frequencies of a predicate in all the websites but also the distribution of the frequencies over the websites. The predicates that appear evenly and frequently in all the websites tend to be more credible. In this regard, we calculate the inter-site frequency of a predicate as follows:

3.2.1 Agreement graph construction.

Given a multi-valued predicate p, we formally define the common values claimed or disclaimed by two sources as inter-source agreement. We consider two-sided inter-source agreements based on mutual exclusion of values. In particular, + agreement, the agreement between two sources (e.g. s₁ and s₂) on their positive claims of p, is denoted as A_p(s₁, s₂) and calculated as:

Similarly, − agreement, the agreement between two sources on their negative claims of p, is denoted as Ã_p(s₁,s₂) and calculated as:

Based on the above quantified inter-source agreements, we construct two fully connected weighted digraphs, i.e. ± agreement graphs. Specifically, in each graph, each vertex denotes a source, and each weighted directed link between two sources represents that one source that agrees with/endorses the other source with an endorsement degree as weight. + Agreement graph models the + agreement among the sources, while − agreement graph focuses on capturing the − agreement among the sources.

To construct the + agreement graph, we first formalize the endorsement degree between two sources [denoted as A(s₁,s₂)] as the average fraction of values of a source endorsed by the other source over all their shared predicates, which is calculated as follows:

Similarly, we create the − agreement graph by applying the equations as follows:

For both graphs, we normalize the weights of out-going links from every vertex by dividing the link weights by the sum of all out-going link weights from the vertex. This normalization allows the link weights to be interpreted as the transition probabilities for the random walk computations directly.

3.2.2 Value veracity estimation.

We apply the Fixed Point Computation (FPC) model to capture the transitive propagation of source reliability through endorsement links, based on the above constructed graphs. In particular, we consider each graph as a Markov chain, where each vertex serves as a state and each link weight serves as the probability of transition between the linked two states. We then compute the asymptotic stationary visiting probabilities of the Markov random walk. For each graph, the calculated probabilities sum up to 1, implying that they cannot be interpreted as the positive or negative precision of each source directly. However, this normalized feature renders the probabilities of each source in both graphs comparable. Also, the ranking of visiting probabilities of each graph implies the ranking of source precision. Moreover, the visiting probabilities capture the following two features:

1. Vertices with more input links that have weights bigger than η are assigned with higher visiting probabilities, because those sources are endorsed by a larger number of sources and should be more trustworthy.

2. Endorsement from a source with more input links that have weights bigger than η, should be more trusted than that from other sources, because authoritative sources tend to be of higher quality and the sources endorsed by authoritative sources tend to be more trustworthy as well.

Based on the above analysis, we refer to the visiting probability of each source in the + agreement (resp., − agreement) graph as a vote for its positive (resp., negative) claims being true (resp., false). Therefore, we estimate the veracity of each possible value of a predicate by:

Algorithm 2 demonstrates the detailed procedure of our approach, which has a time complexity of O(|S|² + |V|). Note that, there are many mature distributed computing tools that can be used for random walk computation to reduce the time complexity. For example, Apache Hama[7] is a framework for Big Data analytics, which uses the Bulk Synchronous Parallel (BSP) computing model. It includes the Graph package for vertex-centric graph computations. We can easily extend the Vertex class to create a class for realizing parallel random walk computation.

4.1.1 Existing KB extractor.

Because Freebase and DBpedia are relatively high-quality resources, we did not use precision to measure the predicate extractions, but simply regarded all the extractions are reasonable predicates. We first used the inheritance relations between types and their sub-types to conduct predicate extraction on each KB separately. Then, our extractor combined the extractions from both KBs. Table II shows the extraction results, indicating that combining the predicates of different KBs significantly increases the number of the predicates for each type.

4.1.2 Query stream extractor.

We collected a query stream with 29,283,918 query records, which is the combination of two real-world data sets, namely, Google[8] and AOL[9]. To further study the distribution of precision regarding the extracted predicates, we ranked the predicates of each type by the EntityDiversity (T, p) of each predicate, and evaluated the precision of top-k (k = 10, 20, 50, 100) predicates for each type. Table III shows the empirical results regarding our query stream extractor, implying that the quality of extractions in terms of number and precision depends on both the number of relevant query records and the natural features of the type. In particular, as type Country and type Film both have relatively large sets of relevant query records, the precision of extractions for either of them is higher than that of other representative types. However, as type Country inherently has more predicates concerned by users, type Country had more credible predicates (i.e. 182) extracted from a query stream than type Film. On the other hand, type Hotel has only 15, 544 relevant query records. Thus, no reasonable predicate could be found for it. For all the types, the precision of the top-k predicates peaked at k = 10, but decreased as k increased. This is consistent with our assumption that the predicates appear with various entities would be more credible. Although the results showed good precision (60-100 per cent), the query stream used for the experiments is still relatively small. It is reasonable to anticipate that more high-quality predicate extractions can be obtained if a larger query stream is available.

4.1.3 DOM tree extractor.

We first merged the extractions obtained from the above two extractors to construct the seed predicate set. Duplication removal was also conducted during this procedure. For example, for type Book, which has 96 predicate extractions from query stream and 60 predicates extracted from existing KBs, 118 seed predicates were finally obtained. Induced by the seed predicates, we exploited crawler4j[10] to crawl the Web and jsoup 1.8.1 to reformat the collected Web pages. In particular, we collected 5 representative websites for each target type, and for each website, we crawled 100 representative Web pages. We filtered out all the nodes with long text (more than ten words in our case) to avoid tackling too many non-predicate nodes. Both the inter-site feature filter and intra-site feature filter were applied to further refine the extractions. Table IV shows the extraction results from DOM trees. For all the five representative types, more predicates were extracted from DOM trees than from either query stream or existing KBs. Also, more seeds tended to lead to more predicates extracted from DOM trees. The results also show the high precision achieved by our extractor with precision that ranges from 79.8 to 93.3 per cent. This is due to the phenomenon that the data contained in DOM trees are often structured and clean. Our empirical studies validate that DOM trees are high-quality resources for predicate extraction, which, unfortunately, are not considered in many recent research efforts regarding KB construction such as Biperpedia (Gupta et al., 2014).

4.2.1 Experimental setup.

We used the following two real-world data sets in our experiments. Each predicate in both data sets is multi-valued:

1. Book-Author Data set (Yin et al., 2008). This data set is collected by crawling www.abebooks.com, which contains 33, 971 data records provided by numerous book stores (i.e. sources). Each record represents the positive claims provided by a specific source regarding the author list of a specific book (i.e. predicate). The ground truth contributed within the original data set was used as the gold standard. We conducted duplication removal to make the problem more challenging – otherwise, even a naive approach could return relatively high-quality results. Finally, we obtained 12, 623 distinct claims, where 649 sources provide author names on 664 books, and each book has 3.2 authors on average.

2. Parent–Children Data set (Pasternack and Roth, 2010). This data set contains 11,099,730 records about individuals’ dates of birth, dates of death and/or the names of their parents/children and spouses. These records were collected from Wikipedia and were edited by numerous users (i.e. sources). We extracted the latest editing records from the data set as the ground truth for experimental purposes. For the sake of validating our multi-valued truth discovery approach, we specially extracted the records on the parent–children relations from the data set. We also removed the duplicated records for this data set, and finally, we obtained 55,259 sources claiming children for 2,579 people, where each person has 2.45 children on average.

To conduct comprehensive comparison studies, we selected three types of representative truth discovery methods as baselines.

4.2.2 Comparison studies.

Table V shows the performance of different approaches on the two data sets in terms of accuracy and efficiency (i.e. precision, recall, F₁ score and execution time). The results show that our approach consistently performed well: it achieved the best recall and F₁ score among the methods. When compared with the two existing MTD methods (LTM and MBM), our approach required the lowest execution time. This is because LTM conducted complicated Bayesian inference over a probabilistic graphical model, and MBM included time-consuming copy detection. All the algorithms achieved lower precision on the Book–Author data set. The possible reasons include the small scale of this data set, missing values (i.e. true values that are missed by all the data sources) and poor quality of sources, leading to insufficient evidence to support obtaining all correct values. The majority of methods showed higher precision than recall, reflecting relatively high positive precision than negative precision of most real-world sources.

Specifically, Voting achieved relatively lower recall on both data sets among the five STD methods, i.e. the second worst on Book–Author data set and the worst on Parent–Children data set. This is because Voting ignores the differences of source quality and simply determines the truth of data by tuning the predefined threshold. To obtain the nearly perfect precision, the threshold of Voting is set as a high value bigger than 0.5. This result implies that instead of applying for solving multi-valued data fusion problem, Voting can be better used for generating the ground truth for semi-supervised truth-finding approaches. The improved STD methods commonly performed worse than their original versions with lower precision and recall, implying that most real-world sources tend to be cautious, making the predicted number of true values for each predicate smaller than it should be. Besides our approach, 2-Estimates and MBM performed better than other methods. This can be attributed to their consideration of mutual exclusion. Though LTM also takes this implication into consideration, it makes strong assumptions on the prior distributions of latent variables. Once the data set does not comply with the assumed distributions, it is likely to perform poorly. Although our approach achieved no significantly superior precision, the recall was improved drastically, resulting in highest F₁ scores for both data sets. The results reveal that our approach achieves the best overall performance among all the baseline methods, which is consistent with our expectation, because it makes no prior assumption and considers the endorsement relations among sources by combining with the graph-based method.

We also investigated the performance of our approach by tuning the values of the source confidence factor θ from 0 to 1 on both data sets. Figure 2 shows the experimental results in terms of precision, recall and F₁ score on Book–Author data set. The experimental results on Parent–Children data set showed similar conclusions. When θ equaled to 0, indicating that the negative claims were not trusted at all, all the positive claims were labeled as true. In this case, the precision was undoubtedly very low (0.49), as there should be a large number of low-quality sources providing false values; meanwhile, the recall was not surprisingly high (0.84), as all claimed values were regarded as true. The recall was less than 1, implying that some true values were missing and not claimed by any sources. As θ grew, the precision dramatically increased (from 0.49 to 0.83) while the recall slightly decreased (from 0.84 to 0.67), implying that by putting more confidence on source negative precision, our approach would reject more false values than true values. The overall performance peaked at the point of θ = 0.6 with an F₁ score of 0.79, which is consistent with our intuition that source confidence on positive claims should be more respected. For θ ∈ [0.3,0.9], the lowest F₁ score is 0.76, which is still higher than the baselines.