Solar Radio Burst Detection Based on the MobileViT-SSDLite Lightweight Model

We utilized publicly available radio spectrogram data from the e-CALLISTO network (Monstein et al. 2023) to establish our data set. The e-CALLISTO network combines all Compound Astronomical Low-frequency Low-cost Instrument for Spectroscopy and Transportable Observatory (CALLISTO) spectrometers. CALLISTO can continuously observe the solar radio spectrum for 24 hr day^–1 throughout the year. Its main applications include observing solar radio bursts. The data obtained from CALLISTO are FITS files with up to 400 frequencies per sweep. Our process of producing spectrogram images is as follows.

First, CALLISTO solar spectrogram FITS files were downloaded from the e-CALLISTO website. The FITS file is composed of four parts: the ASCII-format header, the binary spectrum, and two binary tables. One table is for the time axis, and the other is for the frequency axis. Then, the pyCALLISTO ⁶ (Pawase & Raja 2020) tool was used to parse the FITS files into radio spectrogram images with background subtraction. Finally, based on the event record information provided by the system, we annotated the events with position and category information, constructing an object detection data set in the same format as COCO (Lin et al. 2014). Table 1 presents the number of instances for each type in the data set.

Type I bursts are not included in the data set because they consist of short-duration bursts superimposed on a stable or slowly varying continuous background, making it difficult to collect and annotate resources from the database. Additionally, since there can be more than one burst event on the same spectrogram image, the number of burst instances exceeds the number of spectrogram images.

Furthermore, based on the classification and burst catalog provided by the e-CALLISTO website, ⁷ type III bursts have been further divided into individual (type III) and grouped (type IIIs) bursts, as shown in Figure 1.

Zoom In Zoom Out Reset image size

Due to the limited overall quantity of the obtained data set and the homogeneous backgrounds of various classes, we employed data augmentation techniques to diversify the training samples and improve the robustness and detection accuracy of the model. The augmentation techniques used include random flipping, random cropping with minimum intersection over union (IoU), and photometric distortions, as shown in Figure 2.

Zoom In Zoom Out Reset image size

In Figure 2, (1) RandomFlip provides the model with invariance to mirror image reflections. (2) MinIoURandomCrop involves randomly selecting a target in the image and calculating its IoU with other targets. Then, based on a predefined IoU threshold, whether to keep the target is randomly determined. If it is retained, a randomly cropped image patch of a certain size is obtained around the target as a training sample. (3) PhotoMetricDistortion (PMD) techniques typically involve random perturbations of brightness, contrast, saturation, hue, and random color space transformations. By applying PMD techniques, more diverse training samples can be generated.

By employing a combination of various data augmentation techniques, we can generate richer and more diverse training data for the radio spectrum, thereby improving the model's generalization ability and robustness and enhancing its performance. It is important to note that the augmented images are used only for training purposes; experiments on real spectrogram images are conducted during testing.

While the vision transformer (ViT; Dosovitskiy et al. 2020) architecture has great potential in computer vision, it has certain drawbacks, such as significant model parameters, high computational demands, lack of spatial inductive biases, and difficulties in training, requiring more training data and iterations. To address these issues, MobileViT introduces several modifications to the ViT architecture. It incorporates depthwise separable convolution (DSC), fewer transformer layers, and feature pooling layers, reducing the model's parameter count and computational costs. Additionally, MobileViT adopts a hybrid architecture that combines a CNN and transformer, benefiting from the lightweight and efficient nature of CNNs and the self-attention mechanism and global receptive field of transformers. Extensive experiments on multiple benchmark data sets have demonstrated the effectiveness of MobileViT in various computer vision tasks, including image classification and object detection.

MobileViT leverages the strengths of both CNN and ViT, enabling it to learn better representations with fewer parameters and more straightforward training configurations. It consists of two main components: MobileViTBlock and inverted residual structures, as illustrated in Figure 4.

Zoom In Zoom Out Reset image size

The MobileViTBlock models local and global information using an input tensor with fewer parameters. It employs standard convolutional layers and pointwise convolutions to capture local representations. The global representations are then established through transformers, utilizing the self-attention mechanism to establish long-range receptive fields. As a result, MobileViT possesses a global receptive field, as shown in Figure 5. In Figure 5, each pixel can see other pixels within the MobileViTBlock. For example, the transformer processes the green pixel using the white pixel (corresponding position in other blocks). Since the white pixel has already encoded information about adjacent pixels using convolutions, the green pixel effectively encodes information from all pixels in the image. The black box represents a block, and the gray grid represents a pixel.

Zoom In Zoom Out Reset image size

The inverted residual structure (MV2 architecture in Figure 3) is a bottleneck depth-separable convolution with residuals, which was introduced in MobileNetV2 (Sandler et al. 2018). It begins with convolutional operations on low-dimensional feature maps and upsampling through channel expansion. Finally, the channel dimension is adjusted to the target value through convolution. In the inverted residual structure, in addition to regular convolutional layers, it includes DSC. The computational cost of DSC is approximately one-eighth to one-ninth that of standard convolution, making it an important component in lightweight convolutional neural networks. DSC decomposes the convolution operation into two steps: depthwise and pointwise convolutions. Depthwise convolution involves applying different convolution kernels to each channel of the input feature map, while pointwise convolution further processes the feature map along the channel dimension using convolution. DSC achieves similar results as standard convolutional layers with fewer parameters and computations.

Due to the significant size differences among different types of solar radio bursts in the spectrogram, we consider using a detection head that can adapt to various scales while meeting real-time requirements.

SSD is a one-stage detection model based on anchor boxes (prior boxes). In general, multiple anchor boxes with different scales or aspect ratios are set for each unit, and the predicted bounding boxes are based on these anchor boxes, which reduces the training difficulty to some extent. By using large-scale feature maps to detect small objects and small-scale feature maps to detect large objects, SSD can handle the detection of both large and small targets. This SSD characteristic makes it well suited for different types of solar radio bursts.

The SSD model achieves high accuracy but has a higher computational cost and model size. SSDLite is a variant of SSD that improves the convolutional layers of the SSD model by replacing them with lightweight modules, achieving a better balance between accuracy and speed.

The original SSD model utilized the Visual Geometry Group (VGG; Simonyan & Zisserman 2014) as the feature extraction network. However, the VGG model has many parameters and is unsuitable for real-time detection. In the SSDLite model, a lightweight model called MobileNetv2 (Sandler et al. 2018) is employed as the feature extractor. It uses smaller default boxes and fewer feature map layers. The SSDLite model is illustrated in Figure 6.

Zoom In Zoom Out Reset image size

In Figure 6, many SSDLite components are similar to SSD. The main difference is the split of the convolution operation into two parts: depthwise convolution and pointwise convolution, represented by the DSC layer in the diagram. This split reduces the computational cost and number of parameters. The SSDLite model has demonstrated excellent detection performance on various benchmark data sets while significantly reducing the model size and computational costs. Consequently, the SSDLite model is widely adopted for object detection tasks on mobile and embedded devices, providing high practical value.

The SSDLite loss function consists of a weighted sum of the classification and localization losses, as shown in Equation (1):

$$\begin{eqnarray}&&L(x,c,l,g)=\displaystyle \frac{{L}_{\mathrm{conf}}(x,c)+\alpha * {L}_{\mathrm{loc}}(x,l,g)}{N}.\end{eqnarray} \tag{ 1 }$$

In the formula, L_conf represents classification loss, L_loc represents localization loss, N represents the number of positive samples of the prior boxes, c denotes the predicted class confidence, l denotes the predicted location values for the corresponding bounding boxes of the prior boxes, g represents the position parameters of the ground-truth boxes, and α is set to 1, similar to SSD (Liu et al. 2016).

SSDLite employs the softmax loss function for the classification loss, which calculates the difference between the predicted class probabilities for each prior box and the actual class. For the localization loss, SSDLite uses the smooth L1 loss function. However, smooth L1 loss may lead to unstable training when dealing with many negative samples. Therefore, our approach replaces the localization loss with a loss function based on IoU.

Continuous research shows several types of localization loss based on IoU, including GIoULoss, DIoULoss, and CIoULoss (Zheng et al. 2019). GIoULoss better measures the overlap between predicted and ground-truth boxes and penalizes the nonoverlapping regions more effectively. DIoULoss and CIoULoss converge faster than IoU loss and GIoULoss, with CIoULoss demonstrating better convergence speed and localization accuracy due to its consideration of comprehensive geometric factors, including overlap area, center point distance, and aspect ratio.

Compared to smooth L1 loss, the localization loss function based on CIoU is more stable and accurate, leading to improved model performance. The CIoULoss we use is given as

$$\begin{eqnarray}&&\begin{array}{l}{L}_{\mathrm{CIoU}}=1-\mathrm{IoU}+\displaystyle \frac{{\rho }^{2}(l,g)}{{C}^{2}}+\alpha \nu \\ \alpha =\displaystyle \frac{\nu }{(1-\mathrm{IoU})+\nu },\nu =\displaystyle \frac{4}{{\pi }^{2}}\left(\arctan \displaystyle \frac{{w}^{{gt}}}{{h}^{{gt}}}-\arctan \displaystyle \frac{w}{h}\right),\end{array}\end{eqnarray} \tag{ 2 }$$

where ρ²(l, g) represents the Euclidean distance between the center points of the predicted box l and the ground-truth box g, C² represents the diagonal Euclidean distance of the minimum enclosing region that can simultaneously contain the predicted and ground-truth boxes, ν is used to measure the aspect ratio consistency between the predicted box and the target box, and α is a parameter used to balance the ratios. The loss function tends to optimize toward increasing the overlap area, especially when the IoU is zero.

Figure 7 illustrates the principle of CIoULoss, which introduces a penalty term for the aspect ratio of the predicted and ground-truth bounding boxes on top of DIoULoss, making the loss function pay more attention to the shape of the bounding boxes. Figure 7(a) shows DIoULoss, where the green box represents the ground-truth box, the yellow box is the predicted box, and the gray box is the minimum enclosing region. Compared to GIoULoss, DIoULoss can measure the distance between the two when the target box encompasses the predicted box. However, in the case shown in Figure 7(b), when the predicted box is inside the target box and multiple predicted boxes have the same center point position, DIoULoss cannot differentiate the positions of these two predicted boxes because it does not consider different aspect ratios.

Zoom In Zoom Out Reset image size

All experiments were conducted on a machine with the following hardware configuration: i7-10700k CPU, NVIDIA GeForce RTX 2080 Ti GPU, and 32 GB of memory. The implementation was performed using PyTorch 1.8.1 (Paszke et al. 2019) and MMDetection v2.15.0. The initial learning rate was set to 0.015, and the optimizer used was SGD with a momentum parameter of 0.9 and a weight decay of 0.0004. For the learning rate decay strategy, we employed the cosine decay learning rate method and set the training epochs to 110 to gradually reduce the learning rate and increase model stability.

The detection accuracy metric used was AP50, where AP refers to the average precision, which is the average accuracy for each class in multiclass predictions. AP50 specifically represents the area under the precision-recall curve when the IoU value is 0.5. It comprehensively considers the impact of accuracy and recall, reflecting the model's performance in recognizing a specific class. The calculation methods for accuracy and recall are as follows:

$$\begin{eqnarray}&&\begin{array}{l}\mathrm{precision}=\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}},\\ \mathrm{recall}=\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}},\end{array}\end{eqnarray} \tag{ 3 }$$

where TP represents the number of detection boxes with IoU > 0.5, FP represents the number of detection boxes with IoU ≤ 0.5 or the number of redundant detection boxes for the same ground truth (GT), and FN represents the number of GTs that were not detected.

The detection speed metric is measured in FPS, which evaluates the speed of object detection. It indicates the number of spectrogram images that can be processed per second. FLOPs can be used to measure the complexity of an algorithm and are commonly used as an indirect measure of the speed of neural network models. The parameter count (Params) refers to the total number of trainable parameters in the model and is used to assess the spatial complexity of the model.

In the experiments, we utilized MobileViT as the feature extraction network, SSDLite as the object detection head, and cross-entropy loss as the classification loss. We replaced the original localization loss (smooth L1 loss) with CIoULoss and kept the cross-entropy loss for the classification loss. Multiple data augmentation strategies were employed to enhance the robustness of the model. The performance of the improved model was evaluated using AP50 for accuracy and FPS for speed, and a comparative experiment was conducted to analyze the effectiveness of the improvements.

We trained the model on the created solar radio spectrogram data set with the aforementioned experimental settings and model architecture. Figure 8 displays the changes in training loss and validation detection accuracy (AP50). The training loss curve reflects a gradual decrease and convergence of the loss, while the AP50 curve demonstrates a gradual increase and stabilization of the model's detection accuracy.

Zoom In Zoom Out Reset image size

To evaluate the model's detection accuracy, we used AP50 as the metric and compared it with several classic models. The comparison included YOLOv3 (Redmon & Farhadi 2018) with MobileNetv2 as the backbone network, YOLOv5s with K-means anchors (referred to as YOLOv5s[a] in Table 2), and YOLOv5s without K-means anchors, using default sizes (referred to as YOLOv5s[b] in Table 2). We also compared our model with the generalized focal loss (GFL; Li et al. 2020) method combined with adaptive training sample selection and SSDLite with SwinTransformer Tiny as the backbone network. The experimental results and comparisons are summarized in Table 2.

In Table 2, our improved model achieved higher average detection accuracy, especially for the less frequent radio burst type IV, where the detection performance was significantly better than other methods. It also outperformed other methods in burst type II. Nevertheless, it slightly underperformed compared to GFL in burst types IIIs and V. Since minimizing the omission of solar radio bursts in this application is crucial, we are particularly interested in the model's recall rate. Our improved model achieved a recall rate of 92%, which was higher than that of the three methods in the YOLO series but lower than that of the GFL model. This difference may be attributed to the fact that the GFL method learns a joint representation of classification scores and localization quality and models the general distribution of predicted box positions, making it more suitable for imbalanced samples.

The comparison of detection speeds among different models can be found in Table 2. The FLOPs of our model are 2.9 G, and the model parameters amount to 5.32 M, indicating a low model complexity suitable for devices with lower computational performance. Due to the presence of self-attention in the computation process of MobileViT, the detection speed of our model was slower than that of the MobileNetV2 backbone network. However, it achieved improved detection accuracy. Compared with SwinTransformer Tiny, which also uses self-attention, it reduces the number of model parameters and improves the detection speed. Compared to the GFL detection network using ResNet50 as the backbone and FPN for feature fusion, our model shows improvements in both speed and accuracy. Compared to the lightweight versions of YOLOv3 (with MobileNetV2 as the backbone) and YOLOv5s, our model improved accuracy and detection speed. However, it required a longer convergence time during training due to SSD's denser set of prior boxes compared to YOLO.

We conducted experiments comparing the MobileViT-SSDLite and MobileNetV2-SSDLite detection networks on solar radio spectrogram images, and the results are shown in Table 3.

According to the experimental results in Table 3, it was found that the use of the MobileViT feature extraction network not only improved the overall detection accuracy but also significantly enhanced the detection performance of the rare class, type IV solar radio bursts. The average AP50 for all burst types increased by 2.1%, while type IV bursts saw a 9.4% improvement. Furthermore, the experiments conducted heat map analysis using GradCAM++ (Chattopadhay et al. 2018) to examine the attention distribution of the two models. GradCAM++ reflects the detection output's sensitivity to the input image's pixel values. The higher the temperature, the more sensitive the region, and the lower the temperature, the less sensitive the region. As shown in Figure 9, MobileViT focused more on the radio bursts' frequency and time domain drift features.

Zoom In Zoom Out Reset image size

In addition to the MobileViT feature extraction network analysis, a comparative experimental analysis was conducted on the data augmentation and localization loss improvements proposed in this paper. The results are shown in Table 4, which demonstrates the effectiveness of these improvements in further enhancing the detection accuracy of the model.

4.4.1. Data Augmentation Improves Performance

Assume data augmentation is not performed, and the solar radio spectrum images are directly used for training, as illustrated in Figure 10(a). In that case, the validation accuracy peaks after a few epochs, achieving a maximum mAP@50 of only 62%. Subsequently, the accuracy declines, indicating that the model is overfitting. We implemented online data augmentation techniques on the data set to address this issue. The data augmentation strategies include RandomFlip, MinIoURandomCrop, and PMD, as presented in Figure 2. These techniques played a crucial role in enhancing detection accuracy. We conducted further experiments on the role of RandomFlip. As shown in Figure 10(b), we compared the data augmentation using only PMD and MinIoURandomCrop. The addition of RandomFlip can improve detection accuracy. By incorporating scale and color transformations and flipping into the training process, the model becomes more resilient to variations in the spectrum images.

Zoom In Zoom Out Reset image size

Furthermore, it learns features invariant to different image transformations, enhancing the model's generalization capability. Data augmentation mitigates overfitting and enables the model to adapt to the diverse range of solar radio spectrum images. This ultimately leads to improved performance and enhanced generalization of the model.

4.4.2. Comparison of Localization Loss Functions

Smooth L1 loss suffers from issues of variable independence and lack of scale invariance. To address this, we replaced the localization loss with the IoULoss series and conducted comparative experiments with GIoULoss and CIoULoss, as shown in Figure 11. The experiments demonstrate that CIoULoss achieves better detection accuracy, enables more stable model training, and improves detection accuracy with a 1.1% increase in AP50.

Zoom In Zoom Out Reset image size

As mentioned previously, we enhanced the feature extraction network, employed data augmentation techniques, and utilized an improved localization loss to improve the detection network. We applied this improved model to detect solar radio burst images. Figure 12 illustrates the visualized detection results for different types of solar radio bursts, including types II, III, IIIs, IV, and V. The blue boxes represent the ground-truth annotations, while the orange boxes indicate the detection results, providing both localization and classification information for the radio bursts. Our detection method improves the accuracy of the bounding boxes and reduces the occurrence of redundant boxes, particularly when the spectral features have a wide range.

Zoom In Zoom Out Reset image size

The lightweight model MobileViT-SSDLite employed in this paper has achieved good results in detecting fine structural features of radio types and quickly processing many radio spectrum images. It contributes to the automation of solar radio spectrum detection research and enables efficient, real-time, and accurate detection. During the construction of the solar radio spectrum object detection data set, a small portion of incorrect annotations occurred, as shown in Figure 13, where type III was erroneously labeled as types IIIs and V. However, based on the visualized detection results, the detection model made correct judgments, indicating that our detection model performs well in radio detection and can provide reference annotation information for further improvements in solar radio spectrum data set establishment.

Zoom In Zoom Out Reset image size

While collecting radio burst data, this paper did not gather noise data specifically for annotation purposes. As a result, the model incorrectly identified certain spectrum images that resembled burst types as radio bursts, as depicted in Figure 14. Interferences originating from the observation instrument can be present and exhibit distinct radio spectrum shapes. These interferences may resemble radio bursts in the spectrum images, leading to erroneous detection outcomes. However, the absence of annotated noise images and a limited number of mislabeled samples have impacted the performance of the quantified results. In most cases, the misclassifications were attributed to background types. It is worth noting that certain noise patterns may resemble radio bursts and necessitate further research to enhance detection accuracy.

Zoom In Zoom Out Reset image size