Improving feature matching strategies for efficient image retrieval

Highlights

• Twin Feature is further discussed and optimized into binary code to reduce memory requirement.

• Dynamic normalization is designed to adjust the norm of image when calculating similarity scores.

• An effective image retrieval framework based on SMK with the proposed methods integrated is designed.

Abstract

A number of state-of-the-art image retrieval systems have been built upon non-aggregated techniques such as Hamming Embedding (HE) and Selective Match Kernel (SMK). However, the retrieval performances of these techniques are directly affected by the quality of feature matching during the search process. In general, undesirable matched results appear mainly due to the following three aspects: (1) the locality of local features, (2) the quantization errors and (3) the phenomenon of burstiness. In this paper, starting from the framework of SMK, an in-depth study of the integration of Twin Feature (TF) and Similarity Maximal Matching (SMM) is fully investigated. To be specific, two effective modifications based on TF and SMM are proposed to further improve the quality of feature matching. On one hand, the original float vectors of TF are replaced with efficient binary signatures, which achieve relatively high efficiency and comparable accuracy of retrieval. On the other hand, Dynamic Normalization (DN) is designed to effectively control the impact of penalization generated by SMM and improve the performance with almost no extra cost. At last, an efficient image retrieval system is designed and realized based on a cloud-based heterogeneous computing framework through Apache Spark and multiple GPUs to deal with large-scale tasks. Experimental results demonstrate that the proposed system can greatly refine the visual matching process and improve image retrieval results.

Introduction

Due to the popularity of social networks, images and videos can be captured and shared easily, leading to an explosive growth of multimedia content. As a result, a number of related applications and techniques have been developed to store and process the immense multimedia data. Among all, content-based image retrieval is one of the most challenging tasks [1]. Given a query image, the aim of such systems is to return a series of relevant images from a huge image repository.

Recent years have witnessed a promising progress on image retrieval. Most image retrieval frameworks are built upon local feature based image retrieval models such as Bag-of-Words (BoW) [2] and Selective Match Kernel (SMK) [3] due to their effectiveness and efficiency. In such models, local features are firstly detected from small patches of an image. These patches are of different sizes and contain abundant information of the image. Then, robust algorithms are applied to mine and encode the structure of patches into feature descriptors. Usually, a large amount of descriptors are used to generate a visual vocabulary and the image can be then converted into several feature vectors by quantizing its descriptors to the visual vocabulary. Thanks to the inverted file indexing structure [4], fast retrieval can be obtained when dealing with a large amount of images.

Actually, one of the most crucial steps to impact the retrieval performance is feature matching within inverted files, which determines the matching score between two images directly. Unlike words in text, visual features are usually represented by numerical vectors, and do not have particular meanings like bird, flower or tower, making it challenging to be matched as precisely as text words. False positive matches appear inevitably even though advanced algorithms are applied mainly due to the following issues.

The first one is the locality of features. Local patches of different semantics may be described similarly and would be assigned to the same visual word, which will dramatically affect the visual matching accuracy. Fig. 1 shows a typical example of false match between two images, where the corresponding pair of features are so similar that even human beings are likely to feel confused if only limited local patches (within circles) are concerned. The second issue is quantization error. Irrelevant descriptors may be assigned to the same visual word when the vocabulary size k is small. On the contrary, similar descriptors may be assigned to different visual words when k is very large. Both of these cases will reduce the discriminative power of local descriptors and thus corrupt the retrieval accuracy. The third one is the burstiness [5] phenomenon. Due to the properties of local features as well as the retrieval model, a visual element may appear more than once in an image which causes multiple matches and impacts the ranking scores of candidate images.

As might be expected, this work is committed to investigating how to enhance the performance of image retrieval through refining the process of feature matching. Three major contributions are proposed as follows:

• The technique of Twin Feature (TF) [6] is reviewed and further studied to become more efficient with binary signatures, which retains the favorable structure of inverted files, and provides higher retrieval efficiency.

• A technique named as Dynamic Normalization (DN) is designed to effectively control the impact of penalization generated by Similarity Maximal Matching (SMM) [6] to resolve burstiness and thus improve the final retrieval accuracy.

• An effective image retrieval system based on the state-of-the-art SMK [3] with TF, SMM and DN integrated is designed for image retrieval.

Generally speaking, the aim of this work is to further improve feature matching for image retrieval and can be considered as an extension of our previous work [6]. Both the methods TF and SMM are further discussed and improved. Moreover, to meet the needs of escalating computational demands, the cloud-based heterogeneous computing framework for large-scale image retrieval [7] is employed to carry out the proposed methods. Experimental results on two benchmark image retrieval datasets Oxford5k¹ and Paris6k² demonstrate the effectiveness of proposed methods.

The rest of this paper is organized as follows. Section 2 gives an overview of related works. The details of the proposed image retrieval system are presented in Section 3. In Section 4, the experimental results are presented and analyzed. Finally, Section 5 concludes this paper.