Soft Bigram distance for names matching

Definition of Soft-Bidist distance

Let X and Y be given names represented as sequences of sizes n and m, respectively, Soft-Bidist is defined as follows: (5)${\displaystyle {BIDIST}_{\mathrm{s},\mathrm{t}}\left(\mathrm{i},\mathrm{j}\right)=\left\{\begin{array}{c}\mathrm{Max}\left(\mathrm{i},\mathrm{j}\right)\left(\mathrm{i}=0\text{or}\mathrm{j}=0\right)\hfill \\ \mathrm{Min}\left\{\begin{array}{c}{BIDIST}_{\mathrm{s},\mathrm{t}}\left(\mathrm{i}-1,\mathrm{j}\right)+I{D}_n\left(T_{i,j}^n\right).\hfill \\ {BIDIST}_{\mathrm{s},\mathrm{t}}\left(\mathrm{i},\mathrm{j}-1\right)+I{D}_n\left(T_{i,j}^n\right)\hfill \\ {BIDIST}_{\mathrm{s},\mathrm{t}}\left(\mathrm{i}-1,\mathrm{j}-1\right)+d_n\left(T_{i,j}^n\right).\hfill \\ \hfill \end{array}\right.\hfill \end{array}\right..}$

The distance scale for Soft-Bidist is shown as follows: (6)${\displaystyle {\mathrm{d}}_{\mathrm{n}}\left({\mathrm{T}}_{\mathrm{i},\mathrm{j}}^{\mathrm{n}}\right)=\left\{\begin{array}{c}{\mathrm{wt}}_1,\text{if}\left({\mathrm{S}}_{\mathrm{i}-1}={\mathrm{T}}_{\mathrm{j}-1}\right)\text{and}\left({\mathrm{S}}_{\mathrm{i}}={\mathrm{T}}_{\mathrm{j}}\right)\text{Case}1\hfill \\ {\mathrm{wt}}_2,\text{if}\left({\mathrm{S}}_{\mathrm{i}-1}\ne {\mathrm{T}}_{\mathrm{j}-1}\right)\text{and}\left({\mathrm{S}}_{\mathrm{i}}\ne {\mathrm{T}}_{\mathrm{j}}\right)\text{Case}2\hfill \\ \mathrm{and}\left({\mathrm{S}}_{\mathrm{i}-1}\ne {\mathrm{T}}_{\mathrm{j}}\right)\text{and}\left({\mathrm{S}}_{\mathrm{i}}\ne {\mathrm{T}}_{\mathrm{j}-1}\right)\hfill \\ {\mathrm{wt}}_3,\text{if}\left({\mathrm{S}}_{\mathrm{i}-1}={\mathrm{T}}_{\mathrm{j}}\right)\text{and}\left({\mathrm{S}}_{\mathrm{i}}={\mathrm{T}}_{\mathrm{j}-1}\right)\text{Case}3\hfill \\ {\text{wt}}_4,\mathrm{if}\left({\mathrm{S}}_{\mathrm{i}-1}\ne {\mathrm{T}}_{\mathrm{j}-1}\right)\text{and}\left({\mathrm{S}}_{\mathrm{i}}={\mathrm{T}}_{\mathrm{j}}\right)\text{Case}4\hfill \\ {\mathrm{wt}}_5,\text{if}\left({\mathrm{S}}_{\mathrm{i}-1}={\mathrm{T}}_{\mathrm{j}-1}\right)\text{and}\left({\mathrm{S}}_{\mathrm{i}}\ne {\mathrm{T}}_{\mathrm{j}}\right)\text{Case}5\hfill \\ {\mathrm{wt}}_6,\text{if}\left({\mathrm{S}}_{\mathrm{i}-1}={\mathrm{T}}_{\mathrm{j}}\right)\text{and}\left({\mathrm{S}}_{\mathrm{i}}\ne {\mathrm{T}}_{\mathrm{j}-1}\right)\text{Case}6\hfill \\ {\mathrm{wt}}_7,\text{if}\left({\mathrm{S}}_{\mathrm{i}-1}\ne {\mathrm{T}}_{\mathrm{j}}\right)\text{and}\left({\mathrm{S}}_{\mathrm{i}}={\mathrm{T}}_{\mathrm{j}-1}\right)\text{Case}7\hfill \end{array}\right.}$ (7)${\displaystyle {\text{ID}}_{\mathrm{n}}\left({\mathrm{T}}_{\mathrm{i},\mathrm{j}}^{\mathrm{n}}\right)=\left\{\begin{array}{c}{\mathrm{wt}}_8,\text{if}\left({\mathrm{S}}_{\mathrm{i}-1}={\mathrm{T}}_{\mathrm{j}}\right)\text{and}\left({\mathrm{S}}_{\mathrm{i}}\ne {\mathrm{T}}_{\mathrm{j}-1}\right)\text{Case}8\hfill \\ {\mathrm{wt}}_9,\text{if}\left({\mathrm{S}}_{\mathrm{i}-1}\ne {\mathrm{T}}_{\mathrm{j}}\right)\text{and}\left({\mathrm{S}}_{\mathrm{i}}={\mathrm{T}}_{\mathrm{j}-1}\right)\text{Case 9}\hfill \end{array}\right.}$

To increase the accuracy of identifying the names, there is a need to find the set of weights WT = {wt1; wt2; ...; wt9} of the distance scale of Soft-Bidist. For this, a randomized value is used (Levenshtein, 1966; Al-Hagree et al., 2019a; Al-Hagree et al., 2019b; Kondrak, 2005; Millán-Hernandez et al. , 2019; Rees, 2014).

Finding the Distance Scale for Soft-Bidist

The cases are weighted as symbols. These weights are depend on Tables 1 and 2, which are used to adapt to the operational environment and get highly accurate results in various situations. Therefore, Table 1 contains several different weights. After changing the default values of [0, 1, 1, 0, 1, 1, 1, 1 and 1] with wt1, wt2, wt3, wt4, wt5, wt6, wt7, wt8, and wt9 for all cases respectively, the new weights achieve results similar to that obtained by LD algorithm. Again, other default values have been examined [0, 1, 0, 0, 1, 1, 1, 1 and 1] with for all cases respectively, the new weights achieve results similar to that obtained by the DLD algorithm. Finally, other default values of [0, 1, 1, 0.5, 0.5, 1, 1, 1 and 1] for wt1, wt2, wt3, wt4, wt5, wt6, wt7, wt8, and wt9 for all cases respectively, the new weights achieves results similar to that obtained by the N-DIST algorithm. Based on the previous weight values, new weights were added to Table 2.

The estimated measure

The estimated measure is using the f-measure which is also called f-score. The name matching quality has proven to be effective (Christen, 2006a; Christen, 2006b; Kolomvatsos et al., 2013), which is based on precision and recall. These metrics are used for classification tasks. They compare the predicted class of an item with the actual class, as shown in Table 7. Based on Table 7 and following (Kolomvatsos et al., 2013), precision and recall are defined as:

(8)${\displaystyle \text{Precision}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}}$ (9)${\displaystyle \text{Recall}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}}$

Moreover, the F-measure is defined as the weighted combination of precision and recall. The F-measure is defined by: (10)${\displaystyle \text{F-Measure}=\frac{2.\text{Precision}.\text{Recall}}{\text{Precision}+\text{Recall}}}$

Since F-measure is an accuracy measure between 0 and 1, the higher the values, the better and more accurate are the results. The experiments can be seen in Table 8, the mean of f-measures achieved by the proposed Soft-Bidist algorithm on all instances for the used dataset and the threshold is 0.94, which outperforms the other algorithms. Best results shown boldface and worst results underlined. The thresholds are 0.90, 0.85, 0.80, 0.75, 0.70 and 0.65 of all datasets tested (three English datasets, one Portuguese dataset, three species datasets, three genera datasets, and one Arabic dataset).

Table 9 presents the F1-scores for different scenarios. For the dataset 5 (Portuguese 120 pairs), using different Edit Distance. The best results were retrieved with the threshold values for a correct match of 0.65, 0.70, 0.75, 0.80, 0.85 and 0.90 for LD, DLD, N-DIST, MDLD and Soft-Bidist, respectively (Abdulhayoglu, Thijs & Jeuris, 2016). Table 9 shows F-measure vs. Threshold curves for dataset 5 (Portuguese 120 pairs).

Repeating the previous experiment has been carried based on all Datasets Table 10. The proposed algorithms Soft-Bidist (0,1,0,0.2,0.2,1,1,1 and 1) and Soft-Bidist (0,1,0,0.2,0.2,1,1,0.5 and 0.5) gives more accurate results than the algorithms LD,DLD,N-DIS and MDLD for all datasets as shown in Table 10. The mean of f-measures on all datasets as can be seen in Table 10, which equals 0.97, says that accuracy is almost high and reasonable to trust the results. The best results are shown in bold, and the worst results are underlined. The thresholds are 0.90, 0.85, 0.80, 0.75, 0.70 and 0.65 of all datasets tested (three English datasets, one Portuguese dataset, three species datasets, three genera datasets, and one Arabic dataset).