Fine-grained leukocyte classification with deep residual learning for microscopic images

Highlights

• According to the knowledge of white blood test domain, a leukocyte classifier (Cell3Net) based on deep residual learning theory is constructed.

• Multiple training strategies and properly selected and combined together to make the classifier get better generalization performance.

• A benchmark dataset containing about 100,000 leukocytes belonging to 40 categories is built for the purpose of training and testing.

Abstract

Background and objective: Leukocyte classification and cytometry have wide applications in medical domain, previous researches usually exploit machine learning techniques to classify leukocytes automatically. However, constrained by the past development of machine learning techniques, for example, extracting distinctive features from raw microscopic images are difficult, the widely used SVM classifier only has relative few parameters to tune, these methods cannot efficiently handle fine-grained classification cases when the white blood cells have up to 40 categories.

Methods: Based on deep learning theory, a systematic study is conducted on finer leukocyte classification in this paper. A deep residual neural network based leukocyte classifier is constructed at first, which can imitate the domain expert’s cell recognition process, and extract salient features robustly and automatically. Then the deep neural network classifier’s topology is adjusted according to the prior knowledge of white blood cell test. After that the microscopic image dataset with almost one hundred thousand labeled leukocytes belonging to 40 categories is built, and combined training strategies are adopted to make the designed classifier has good generalization ability.

Results: The proposed deep residual neural network based classifier was tested on microscopic image dataset with 40 leukocyte categories. It achieves top-1 accuracy of 77.80%, top-5 accuracy of 98.75% during the training procedure. The average accuracy on the test set is nearly 76.84%.

Conclusions: This paper presents a fine-grained leukocyte classification method for microscopic images, based on deep residual learning theory and medical domain knowledge. Experimental results validate the feasibility and effectiveness of our approach. Extended experiments support that the fine-grained leukocyte classifier could be used in real medical applications, assist doctors in diagnosing diseases, reduce human power significantly.

Introduction

The ratio of leukocytes (white blood cells) in the blood is usually an indicator of diseases. The leukocyte differential count is used in the process of diagnosing diseases by counting the number and ratio of leukocytes in the blood. White blood cell (WBC) test plays a crucial role in detecting and treating diseases such as leukemia, anemia, etc.

Traditionally, white blood cell classification and counting is primarily achieved through a time consuming and troublesome manual process, during which errors often come up. Therefore, an automated and intelligent cell classification approach is of significance. The samples could be processed continuously or in batches in this way. Due to the intrinsic sophistication of white blood cell classification problem, for example, the morphology and features of leukocytes may vary significantly depending on cell types, no rigid rules which are robust and general enough could be applied in white blood cell category recognition. In previous works, researchers usually tend to conquer the WBC test problem by exploiting machine learning techniques. Bikhet et al. [1] propose a white blood cell shape analysis and classification approach based on the morphological characteristics of their outer contour and nuclei. Denoised gray images are used as input for the classification system, where ten features including the area of the cell, the area of the nucleus, cell circularity etc. are selected and extracted from the original images. 71 cell samples belonging to 5 categories are tested in the experiment. Though this approach considers mainly the contour information on the white blood cells, category distinguishing ability of this approach is not good enough. Sinha et al. [2] develop an automatic system for differential blood count using color images of blood smears. Shape, color and texture features are respectively extracted and tested in this system. Meanwhile various classifiers have been explored on the features sets. In the experiments, the best accuracies are obtained based on approaches using neural networks and support vector machine (SVM). However, though the chosen features are relatively simple, the cells happen to be touching can’t be distinguished by the developed system. Hiremath et al. [3] propose an automatic approach for the differential counting of white blood cell. The geometric features such as area, major axis length, minor axis length, perimeter, circularity, ratio of areas of nucleus and cytoplasm are used for identifying the different types of white blood cells. Namely three leukocyte cell types including lymphocyte, monocyte and neutrophil are classified in this approach. Though only low level geometric information is calculated when comparing different types of white blood cells, necessary pre-processing methods are needed. Moreover, empirically chosen thresholds are applied for classification in this approach, which maybe not robust enough. Khashman et al. [4], [5] investigate three different topologies of conventional neural networks for blood cell type identification. Firstly the global pattern averaging descriptor is used to extract the feature vectors from the cell images; then three neural networks which have different input layers with different numbers of neurons are trained. In the experiments, both of the three neural networks could identify red blood cells, white blood cells and platelets. But this approach doesn’t take the background factors into account, so the recognition effect is not very good in practice. Sabino et al. [6] propose a texture approach to leukocyte recognition. Five texture features including inertia, energy, entropy, correlation and homogeneity are calculated based on gray level co-occurrence matrices (GLCM) to differentiate five types of normal leukocytes and chronic lymphocytic leukemia. The performance of texture features are illustrated through experiments on 730 leukocyte images, but the accuracy seems to be a little low. Huang et al. [7] propose a computer assisted approach for leukocyte nucleus segmentation and recognition in blood smear images. Firstly the leukocyte nucleus is segmented based on Otsu’s method. Then the co-occurrence matrix is used as a textural measure of segmented images. Finally the genetic algorithm based k-means is used to classify five kinds of leukocyte. However, constrained by the past development of machine learning techniques, for example, extracting distinctive features from raw microscopic images are difficult, the widely used SVM classifier only has relative few parameters to tune [8], these approaches are mainly aiming at three classification or five classification of leukocytes. Actually leukocytes could be classified into 40 categories according to their medical characteristics. 40 classification for leukocytes will be more challenging and valuable for clinical diagnosis of diseases.

Deep learning makes machine learning proceed a big step toward its original goal – implementing the true artificial intelligence. The deep learning approach has acquired significant breakthrough for many problems in various domains since 2006, triggered a research boom in AI and machine learning communities. Its about learning multiple levels of representation and abstraction that help to make sense of data such as sound, image and video [9], [10], [11], [12]. Currently many deep learning approaches have been developed to overcome recognition and classification tasks in multiple fields. Most notably, Krizhevsky et al. [13] propose a classical deep learning architecture called AlexNet, and show significant improvements upon previous approaches on the image classification task. Witness on the success of AlexNet, several works are proposed to improve its performance. Among them, five representative works are ZFNet [14], VGGNet [15], ResNet [16], GoogLeNet [17] and DenseNet [18]. In addition, Long et al. [19] train a fully convolutional network (FCN) for object semantic segmentation. The input images are directly predicted to dense label maps. However, these networks are mainly aiming at recognizing objects in natural scenes, can’t be applied into micro situation directly.

In this paper, we propose a finer white blood cell classification approach with deep residual learning. Comparing to the traditional shallow machine learning algorithms, our approach addresses the problem through a simple end-to-end learning manner, avoiding error-prone processes such as denoising, white balance, feature extraction, feature selection etc. Hence the classification and cytometry accuracy could be improved.