Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges

2.1 Local and Edge Computing

We start with an overview of local and edge computing. In local computing (LC), the function \( \mathcal {M}(\mathbf {x}) \) is entirely executed by the mobile device. This approach eliminates the need to transfer data over the wireless channel. However, the complexity of the best-performing DNNs most likely exceeds the computing capacity and energy consumption available at the mobile device. Usually, simpler models \( \hat{\mathcal {M}}(\mathbf {x}) \) are used, such as MobileNet [122] and MnasNet [138], which often have a degraded accuracy performance. Besides designing lightweight neural models executable on mobile devices, the widely used techniques to reduce the complexity of models are knowledge distillation [46] and model pruning/quantization [55, 73], described in Section 3.2. Some of the techniques are also leveraged in SC studies to introduce bottlenecks without sacrificing model accuracy, as will be described in the following sections.

In EC, the input \( \mathbf {x} \) is transferred to the edge server, which then executes the original model \( \mathcal {M}(\mathbf {x}) \). In this approach, which preserves full accuracy, the mobile device is not allocated computing workload, but the full input \( \mathbf {x} \) needs to be transferred to the edge server. This may lead to an excessive end-to-end delay in degraded channel conditions and erasure of the task in extreme conditions. A possible approach to reduce the load imposed to the wireless channel, and thus also transmission delay and erasure probability, is to compress the input \( \mathbf {x} \). We define, then, the encoder and decoder models \( \mathbf {z} = F(\mathbf {x}) \) and \( \hat{\mathbf {x}} = G(\mathbf {z}) \), which are executed at the mobile device and edge server, respectively. The distance \( d(\mathbf {x},\hat{\mathbf {x}}) \) defines the performance of the encoding-decoding process \( \hat{\mathbf {x}} = G(F(\mathbf {x})) \), a metric that is separate from, but may influence, the accuracy loss of \( \mathcal {M}(\hat{\mathbf {x}}) \) with respect to \( \mathcal {M}(\mathbf {x}) \), that is, of the model executed with the reconstructed input with respect to the model executed with the original input. Clearly, the encoding/decoding functions increase the computing load at both the mobile device and edge server side. A broad range of different compression approaches exists ranging from low-complexity traditional compression (e.g., JPEG compression for images in EC [101]) to neural compression models [4, 5, 162]. We remark that while the compressed input data, e.g., JPEG objects, can reduce the data transfer time in EC, those representations are designed to allow the accurate reconstruction of the input signal. Therefore, these approaches may (1) decrease privacy as a “reconstructable” representation is transferred to the edge server [147] and (2) result in a larger amount of data to be transmitted over the channel compared to representation specifically designed for the computing task as in bottleneck-based SC, as explained in the following sections.

2.2 Split Computing and Early Exiting

SC aims at achieving the following goals: (1) the computing load is distributed across the mobile device and edge server and (2) establishes a task-oriented compression to reduce data transfer delays. We consider a neural model \( \mathcal {M}(\cdot) \) with L layers, and define \( \mathbf {z}_{\ell } \) as the output of the \( \ell \)th layer. Early implementations of SC select a layer \( \ell \) and divide the model \( \mathcal {M}(\cdot) \) to define the head and tail submodels \( \mathbf {z}_{\ell }{=}\mathcal {M}_{H}(\mathbf {x}) \) and \( \mathbf {\hat{y}}{=}\mathcal {M}_{T}(\mathbf {z}_{\ell }) \), executed at the mobile device and edge server, respectively. In early instances of SC, the architecture and weights of the head and tail model are exactly the same as the first \( \ell \) layers and last \( L-\ell \) layers of \( \mathcal {M}(\cdot) \). This simple approach preserves accuracy but allocates part of the execution of \( \mathcal {M}(\cdot) \) to the mobile device, whose computing power is expected to be smaller than that of the edge server, so that the total execution time may be larger. The transmission time of \( \mathbf {z}_{\ell } \) may be larger or smaller compared to that of transmitting the input \( \mathbf {x} \), depending on the size of the tensor \( \mathbf {z}_{\ell } \). However, we note that in most relevant applications the size of \( \mathbf {z}_{\ell } \) becomes smaller than that of \( \mathbf {x} \) only in later layers, which would allocate most of the computing load to the mobile device. More recent SC frameworks introduce the notion of bottleneck to achieve in-model compression toward the global task [90]. As formally described in the next section, a bottleneck is a compression point at one layer in the model, which can be realized by reducing the number of nodes of the target layer and/or by quantizing its output. We note that as SC realizes a task-oriented compression, it guarantees a higher degree of privacy compared to EC. In fact, the representation may lack information needed to fully reconstruct the original input data.

Another approach to enable mobile computing is referred to as EE. The core idea is to create models with multiple “exits” across the model, where each exit can produce the model output. Then, the first exit providing a target confidence on the output is selected. This approach tunes the computational complexity, determined by the exit point, to the sample or to system conditions. Formally, we can define a sequence of models \( \mathcal {M}_i \) and \( \mathcal {B}_i, i= 1,\ldots ,N \). Model \( \mathcal {M}_i \) takes as input \( \mathbf {z}_{i-1} \) (the output of model \( \mathcal {M}_{i-1} \)) and outputs \( \mathbf {z}_i \), where we set \( \mathbf {z}_{0}=\mathbf {x} \). The branch models \( \mathcal {B}_i \) take as input \( \mathbf {z}_i \) and produce the estimate of the desired output \( \mathbf {y}_i \). Thus, the concatenation of \( \mathcal {M}_1,\ldots ,\mathcal {M}_N \) results in an output analogous to that of the original model. Intuitively, the larger the number of models used to produce the output \( \mathbf {y}_i \), the better the accuracy. Thus, while SC optimizes intermediate representations to preserve information toward the final task (e.g., classification) for the whole dataset, early-exit models take a “per sample” control perspective. Each sample will be sequentially analyzed by concatenations of \( \mathcal {M}_i \) and \( \mathcal {B}_i \) sections until a predefined confidence level is reached. The hope is that a portion of the samples will require a smaller number of sections compared to executing the whole sequence.

3.1 Lightweight Models

From a conceptual perspective, the design of small deep learning models is one of the simplest ways to reduce inference cost. However, there is a tradeoff between model complexity and model accuracy, which makes this approach practically challenging when aiming at high model performance. The MobileNet series [47, 48, 122] is one among the most popular lightweight models for computer vision tasks, where Howard et al. [48] describe the first version, MobileNetV1. By using a pair of depth-wise and point-wise convolution layers in place of standard convolution layers, the design drastically reduces model size, and thus computing load. Following this study, Sandler et al. [122] proposed MobileNetV2, which achieves improved accuracy. The design is based on MobileNetV1 [48] and uses the bottleneck residual block, a resource-efficient block with inverted residuals and linear bottlenecks. Howard et al. [47] present MobileNetV3, which further improves the model accuracy and is designed by a hardware-aware neural architecture search [138] with NetAdapt [161]. The largest variant of MobileNetV3, MobileNetV3-Large 1.0, achieves a comparable accuracy to ResNet-34 [43] for the ImageNet dataset, while reducing by about 75% the model parameters.

While many of the lightweight neural networks are often manually designed, there are also studies on automating the neural architecture search (NAS) [173]. For instance, Zoph et al. [174] design a novel search space through experiments with the CIFAR-10 dataset [63], which is then scaled to larger, higher-resolution image datasets such as the ImageNet dataset [120], to design their proposed model: NASNet. Leveraging the concept of NAS, some studies design lightweight models in a platform-aware fashion. Dong et al. [23] propose the Device-aware Progressive Search for Pareto-optimal Neural Architectures (DDP-Net) framework, which optimizes the network design with respect to two objectives: device-related (e.g., inference latency and memory usage) and device-agnostic (e.g., accuracy and model size) objectives. Similarly, Tan et al. [138] propose an automated mobile neural architecture search (MNAS) method and design the MnasNet models by optimizing both model accuracy and inference time.

3.2 Model Compression

A different approach to produce small DNN models is to “compress” a large model. Model pruning and quantization [38, 39, 55, 79] are the dominant model compression approaches. The former removes parameters from the model, while the latter uses fewer bits to represent them. In both these approaches, a large model is trained first and then compressed, rather than directly designing a lightweight model followed by training. Jacob et al. [55] empirically show that their quantization technique leads to an improved tradeoff between inference time and accuracy on MobileNet [48] for image classification tasks on Qualcomm Snapdragon 835 and 821 compared to the original, float-only MobileNet. For what concerns model pruning, Li et al. [75] and Liu et al. [86] demonstrate that it is difficult for model pruning itself to accelerate inference while achieving strong performance guarantees on general-purpose hardware due to the unstructured sparsity of the pruned model and/or kernels in layers.

Knowledge distillation [8, 46] is another popular model compression method. While model pruning and quantization make trained models smaller, the concept of knowledge distillation is to provide outputs extracted from the trained model (called “teacher”) as informative signals to train smaller models (called “student”) in order to improve the accuracy of predesigned small models. Thus, the goal of the process is that of distilling knowledge of a trained teacher model into a smaller student model for boosting accuracy of the smaller model without increasing model complexity. For instance, Ba and Caruana [3] propose a method to train small neural networks by mimicking the detailed behavior of larger models. The experimental results show that models trained by this mimic learning method achieve performance close to that of deeper neural networks on some phoneme recognition and image recognition tasks. The formulation of some knowledge distillation methods will be described in Section 4.4.

4.1 Split Computing without DNN Modification

In this class of approaches, the architecture and weights of the head \( \mathcal {M}_{H}(\cdot) \) and tail \( \mathcal {M}_T(\cdot) \) models are exactly the same as the first \( \ell \) layers and last \( L-\ell \) layers of \( \mathcal {M}(\cdot) \). To the best of our knowledge, Kang et al. [60] proposed the first SC approach (called “Neurosurgeon”), which searches for the best partitioning layer in a DNN model for minimizing total (end-to-end) latency or energy consumption. Formally, inference time in SC is the sum of processing time on a mobile device, delay of communication between a mobile device and edge server, and processing time on an edge server.

Interestingly, their experimental results show that the best partitioning (splitting) layers in terms of energy consumption and total latency for most of the considered models result in either their input or output layers. In other words, deploying the whole model on either a mobile device or an edge server (i.e., local computing or EC) would be the best option for such DNN models. Following the work by Kang et al. [60], the research communities explored various SC approaches mainly focused on CV tasks such as image classification. Table 1 summarizes the studies on SC without architectural modifications.

Jeong et al. [58] used this partial offloading approach as a privacy-preserving way for computation offloading to blind the edge server to the original data captured by client. Leveraging neural network quantization techniques, Li et al. [73] discussed best splitting point in DNN models to minimize inference latency and showed that quantized DNN models did not degrade accuracy compared to the (pre-quantized) original models. Choi and Bajić [13] proposed a feature compression strategy for object detection models that introduces a quantization/video-coding-based compressor to the intermediate features in YOLO9000 [115].

Eshratifar et al. [25] propose JointDNN for collaborative computation between the mobile device and cloud and demonstrate that using either local computing only or cloud computing only is not an optimal solution in terms of inference time and energy consumption. Different from [60], they consider not only discriminative deep learning models (e.g., classifiers) but also generative deep learning models and autoencoders as benchmark models in their experimental evaluation. Cohen et al. [16] introduce a technique to code the output of the head portion in a split DNN to a wide range of bit-rates and demonstrate the performance for image classification and object detection tasks. Pagliari et al. [106] first discuss the collaborative inference for simple recurrent neural networks, and their proposed scheme is designed to automatically select the best inference device for each input data in terms of total latency or end-device energy. Itahara et al. [53] use dropout layers [133] to emulate a packet loss scenario rather than for the sake of compression and discuss the robustness of VGG-based models [128] for split computing.

While only a few studies in Table 1 heuristically choose splitting points [13, 16], most of the other studies [25, 58, 60, 73, 76, 106, 168] in Table 1 analyze various types of cost (e.g., computational load and energy consumption on mobile device, communication cost, and/or privacy risk) to partition DNN models at each of their splitting points. Based on the analysis, performance profiles of the split DNN models are derived to inform selection. Concerning metrics, many of the studies in Table 1 do not discuss task-specific performance metrics such as accuracy. This is in part because the proposed approaches do not modify the input or intermediate representations in the models (i.e., the final prediction will not change). On the other hand, Choi and Bajić [13], Cohen et al. [16], and Li et al. [73] introduce lossy compression techniques to intermediate stages in DNN models, which may affect the final prediction results. Thus, discussing the tradeoff between compression rate and task-specific performance metrics would be essential for such studies. As shown in the table, such tradeoff is discussed only for CV tasks, and many of the models considered in such studies have weak performance compared with state-of-the-art models and complexity within reach of modern mobile devices. Specific to image classification tasks, most of the models considered in the studies listed in Table 1 are more complex and/or the accuracy is comparable to or lower than that of lightweight baseline models such as MobileNetV2 [122] and MnasNet [138]. Thus, in future work, more accurate models should be considered to discuss the performance tradeoff and further motivate SC approaches.

4.2 The Need for Bottleneck Injection

While Kang et al. [60] empirically show that executing the whole model on either the mobile device or edge server would be best in terms of total inference and energy consumption for most of their considered DNN models, their proposed approach finds the best partitioning layers inside some of their considered CV models (convolutional neural networks (CNNs)) to minimize the total inference time. There are a few trends observed from their experimental results: (1) communication delay to transfer data from the mobile device to edge server is a key component in SC to reduce total inference time; (2) all the neural models they considered for NLP tasks are relatively small (consisting of only a few layers), which potentially resulted in finding that the output layer is the best partition point (i.e., local computing) according to their proposed approach; and (3) similarly, not only DNN models they considered (except VGG [128]) but also the size of the input data to the models (see Table 2) are relatively small, which gives more advantage to EC (fully offloading computation). In other words, it highlights that complex CV tasks requiring large (high-resolution) images for models to achieve high accuracy such as ImageNet and COCO datasets would be essential to discuss the tradeoff between accuracy and execution metrics to be minimized (e.g., total latency, energy consumption) for SC studies. The key issue is that straightforward SC approaches like Kang et al. [60] rely on the existence of natural bottlenecks—that is, intermediate layers whose output \( \mathbf {z}_{\ell } \) tensor size is smaller than the input—inside the model. Without such natural bottlenecks in the model, straightforward splitting approaches would fail to improve performance in most settings [6, 35].

Some models, such as AlexNet [64], VGG [128], and DenseNet [51], possess such layers [90]. However, recent DNN models such as ResNet [43], Inception-v3 [136], Faster R-CNN [117], and Mask R-CNN [42] do not have natural bottlenecks in the early layers; that is, splitting the model would result in compression only when assigning a large portion of the workload to the mobile device. As discussed earlier, reducing the communication delay is key to minimize total inference time in SC. For these reasons, introducing artificial bottlenecks to DNN models by modifying their architecture is a recent trend and has been attracting attention from the research community. Since the main role of such encoders in SC is to compress intermediate features rather than to complete inference, the encoders usually consist of only a few layers. Also, the resulting encoders in SC to be executed on constrained mobile devices are often much smaller (e.g., 10K parameters in the encoder of ResNet-based SC model [94]) than lightweight models such as MobileNetV2 [122] (3.5M parameters) and MnasNet [138] (4.4M parameters). Thus, even if the model accuracy is either degraded or comparable to such small models, SC models are still beneficial in terms of computational burden and energy consumption at the mobile devices.

4.3 Split Computing with Bottleneck Injection

This class of models can be described as composed of three sections: \( \mathcal {M}_{E} \), \( \mathcal {M}_{D} \), and \( \mathcal {M}_{T} \). We define \( \mathbf {z}_{\ell }|\mathbf {x} \) as the output of the \( \ell \)th layer of the original model given the input \( \mathbf {x} \). The concatenation of the \( \mathcal {M}_{E} \) and \( \mathcal {M}_{D} \) models is designed to produce a possibly noisy version \( \hat{\mathbf {z}}_{\ell }|\mathbf {x} \) of \( \mathbf {z}_{\ell }|\mathbf {x} \), which is taken as input by \( \mathcal {M}_{T} \) to produce the output \( \hat{\mathbf {y}} \), on which the accuracy degradation with respect to \( \mathbf {y} \) is measured. The models \( \mathcal {M}_{E} \), \( \mathcal {M}_{D} \) function as specialized encoders and decoders in the form \( \hat{\mathbf {z}}_{\ell }=\mathcal {M}_{D}(\mathcal {M}_E(\mathbf {x})) \), where \( \mathcal {M}_{E}(\mathbf {x}) \) produces the latent variable \( \mathbf {z} \). In worlds, the two first sections of the modified model transform the input \( \mathbf {x} \) into a version of the output of the \( \ell \)th layer via the intermediate representation \( \mathbf {z} \), thus functioning as encoder/decoder functions. The model is split after the first section; that is, \( \mathcal {M}_{E} \) is the head model, and the concatenation of \( \mathcal {M}_{D} \) and \( \mathcal {M}_{T} \) is the tail model. Then, the tensor \( \mathbf {z} \) is transmitted over the channel. The objective of the architecture is to minimize the size of \( \mathbf {z} \) to reduce the communication time while also minimizing the complexity of \( \mathcal {M}_E \) (that is, the part of the model executed at the—weaker—mobile device) and the discrepancy between \( \mathbf {y} \) and \( \hat{\mathbf {y}} \). The layer between \( \mathcal {M}_E \) and \( \mathcal {M}_D \) is the injected bottleneck.

Table 3 summarizes SC studies with bottleneck-injected strategies. To the best of our knowledge, the papers in [26] and [90] were the first to propose altering existing DNN architectures to design relatively small bottlenecks at early layers in DNN models, instead of introducing compression techniques (e.g., quantization, autoencoder) to the models, so that communication delay (cost) and total inference time can be further reduced. Following these studies, Hu and Krishnamachari [49] introduce bottlenecks to MobileNetV2 [122] (modified for CIFAR datasets) in a similar way for SC and discuss end-to-end performance evaluation. Choi et al. [14] combine multiple compression techniques such as quantization and tiling besides convolution/deconvolution layers and design a feature compression approach for object detectors. Similar to the concept of bottleneck injection, Shao and Zhang [126] find that over-compression of intermediate features and inaccurate communication between computing devices can be tolerated unless the prediction performance of the models is significantly degraded by them. Also, Jankowski et al. [57] propose introducing a reconstruction-based bottleneck to DNN models, which is similar to the concept of BottleNet [26]. A comprehensive discussion on the delay/complexity/accuracy tradeoff can be found in [91, 164].

These studies are all focused on image classification. Other CV tasks present further challenges. For instance, state-of-the-art object detectors such as R-CNN models have more narrow range of layers that we can introduce bottlenecks due to the network architecture, which has multiple forward paths to forward outputs from intermediate layers to the feature pyramid network (FPN) [81]. The head network distillation training approach—discussed later in this section—was used by Matsubara and Levorato [94] to address some of these challenges and reduce the amount of data transmitted over the channel by 94% while degrading mAP loss by 1 point. Assine et al. [2] introduce bottlenecks to the EfficientDet-D2 [139] object detector and apply the training method based on the generalized head network distillation [94] and mutual learning [159] to the modified model. Following the studies on SC for resource-constrained edge computing systems [90, 91, 164], Sbai et al. [124] introduce autoencoder to small classifiers and train them on a subset of the ImageNet dataset in a similar manner. These studies discuss the tradeoff between accuracy and memory size on mobile devices, considering communication constraint-based 3G and LoRa technologies [121]. Similar to [2, 93, 94], Lee et al. [70] design a lightweight encoder for an object detector on the mobile device followed by both a module to amplify the compressed feature and the object detector to be executed on the edge server. Matsubara et al. [92] empirically show that bottleneck-injected models can be further improved by elaborating the methods to train the models. The resulting models outperform models with autoencoder-based feature compression (e.g., Figure 5) in terms of the tradeoff between model accuracy and transferred data size.

Matsubara et al. [97] propose a supervised compression method for resource-constrained edge computing systems, which adapts ideas from knowledge distillation and neural image compression [4, 5]. Their student model (namely, Entropic Student) contains a lightweight encoder with a learnable prior, which quantizes and entropy-codes latent representations under a prior probability model for efficiently saving the size of data to be offloaded to the edge server. By adjusting a balancing weight in their loss function during training, we can control the tradeoff between data size (rate) and model accuracy (distortion). The performance of the entropic student model was demonstrated for three large-scale downstream supervised tasks: image classification (ImageNet), object detection (COCO), and semantic segmentation (COCO, PASCAL VOC). Notably, the representation produced by a single trained encoder of the entropic student model can serve multiple downstream tasks. Following the study, Matsubara et al. [96] further investigate this approach and empirically show that it generalizes to other reference models (e.g., ResNet-101 [43], RegNetY-6.4GF [113], Hybrid ViT [134]). Through experiments, the study also points out that simply introducing such bottleneck layers at later layers in a model can improve the conventional rate-distortion (R-D) tradeoff, which will result in most of the computational load being assigned to a weak mobile device.

In contrast to SC studies without bottlenecks in Table 1, many of the studies on bottleneck injection strategies in Table 3 are published with code that would help the research communities replicate/reproduce the experimental results and build on existing studies.

4.4 SC with Bottlenecks: Training Methodologies

Given that recent SC studies with bottleneck injection strategies result in more or less accuracy loss compared to the original models (i.e., without injected bottlenecks), various training methodologies are used and/or proposed in such studies. Some of the training methods are designed specifically for architectures with injected bottlenecks. We now summarize the differences between the various training methodologies used in recent SC studies.

We recall that \( \mathbf {x} \) and \( \mathbf {y} \) are an input (e.g., an RGB image) and the corresponding label (e.g., one-hot vector), respectively. Given an input \( \mathbf {x} \), a DNN model \( \mathcal {M} \) returns its output \( \mathbf {\hat{y}} = \mathcal {M}(\mathbf {x}) \) such as class probabilities in classification task. Each of the L layers of model \( \mathcal {M} \) can be either low-level (e.g., convolution [69], batch normalization [52], ReLU [100]) or high-level layers (e.g., residual block in ResNet [43] and dense block in DenseNet [51]), which are composed by multiple low-level layers. \( \mathcal {M}(\mathbf {x}) \) is a sequence of the L layer functions \( \mathrm{f}_{j} \)s, and the jth layer transforms \( \mathbf {z}_{j-1} \), the output from the previous \( {(j-1)} \)th layer: (1) \( \begin{equation} \mathbf {z}_{j} = \left\lbrace \!\!\!\begin{array}{ll} \mathbf {x} & j = 0 \\ \mathrm{f}_j(\mathbf {z}_{j-1}, \mathbf {\theta }_j) & 1 \le j \lt L, \\ \mathrm{f}_L(\mathbf {z}_{L-1}, \mathbf {\theta }_L) = \mathcal {M}(\mathbf {x}) = \mathbf {\hat{y}} & j = L \end{array} \right. \end{equation} \) where \( \mathbf {\theta }_{j} \) denotes the jth layer’s hyperparameters and parameters to be optimized during training.

Cross-entropy-based Training

To optimize parameters in a DNN model, we first need to define a loss function and update the parameters by minimizing the loss value with an optimizer such as stochastic gradient descent and Adam [62] during training. In image classification, a standard method is to train a DNN model \( \mathcal {M} \) in an end-to-end manner using the cross-entropy like many of the studies [26, 49, 91] in Table 3. For simplicity, here we focus on the categorical cross-entropy and assume \( c \equiv \mathbf {y} \) is the correct class index given a model input \( \mathbf {x} \). Given a pair of \( \mathbf {x} \) and c, we obtain the model output \( \mathbf {\hat{y}} = \mathcal {M}(\mathbf {x}) \), and then the (categorical) cross-entropy loss is defined as (2) \( \begin{equation} \mathcal {L}_\text{CE}(\mathbf {\hat{y}}, c) = -\log \left(\frac{\exp \left(\hat{\mathbf {y}}_{c} \right)}{\sum _{j \in \mathcal {C}} \exp \left(\hat{\mathbf {y}}_j \right)} \right)\!, \end{equation} \) where \( \hat{\mathbf {y}}_{j} \) is the class probability for the class index j, and \( \mathcal {C} \) is a set of considered classes (\( c \in \mathcal {C} \)).

As shown in Equation (2), the loss function used in cross-entropy-based training methods are used as a function of the final output \( \mathbf {\hat{y}} \), and thus are not designed for SC frameworks. While Eshratifar et al. [26], Hu and Krishnamachari [49], Lee et al. [70], and Shao and Zhang [126] use cross-entropy to train bottleneck-injected DNN models in end-to-end manners (Figure 3), Matsubara et al. [91] empirically show that these methods cause a larger accuracy loss in complex tasks such as the ImageNet dataset [120] compared to other more advanced techniques, including knowledge distillation.

Fig. 3.

View Figure

Knowledge Distillation

Complex DNN models are usually trained to learn parameters for discriminating between a large number of classes (e.g., 1,000 in ImageNet dataset) and are often overparameterized. KNOWLEDGE DISTILLATION (KD) [3, 46, 78] is a training scheme to address this problem, and trains a DNN model (called “student”) using additional signals from a pretrained DNN model (called “teacher” and often larger than the student). In standard cross-entropy-based training—that is, using “hard targets” (e.g., one-hot vectors)—we face a side effect that the trained models assign probabilities to all of the incorrect classes. From the relative probabilities of incorrect classes, we can see how large models tend to generalize.

As illustrated in Figure 4, by distilling the knowledge from a pretrained complex model (teacher), a student model can be more generalized and avoid overfitting to the training dataset, using the outputs of the teacher model as “soft targets” in addition to the hard targets [46]. (3) \( \begin{equation} \mathcal {L}_\text{KD}(\hat{\mathbf {y}}^\text{S}, \hat{\mathbf {y}}^\text{T}, \mathbf {y}) = \alpha \mathcal {L}_\text{task}(\hat{\mathbf {y}}^\text{S}, \mathbf {y}) + (1 - \alpha) \tau ^2 \mathrm{KL} \left(\mathrm{q}(\hat{\mathbf {y}}^\text{S}), \mathrm{p}(\hat{\mathbf {y}}^\text{T}) \right)\!, \end{equation} \) where \( \alpha \) is a balancing factor (hyperparameter) between the hard target (left term) and soft target (right term) losses, and \( \tau \) is another hyperparameter called temperature to soften the outputs of teacher and student models in Equation (4). \( \mathcal {L}_\text{task} \) is a task-specific loss function, and it is a cross-entropy loss in image classification tasks, i.e., \( \mathcal {L}_\text{task} = \mathcal {L}_\text{CE} \). \( \mathrm{KL} \) is the Kullback-Leibler divergence function, where \( \mathrm{q}(\hat{\mathbf {y}}^\text{S}) \) and \( \mathrm{p}(\hat{\mathbf {y}}^\text{T}) \) are probability distributions of student and teacher models for an input \( \mathbf {x} \), that is, \( \mathrm{q}(\hat{\mathbf {y}}^\text{S}) = [\mathrm{q}_{1}(\hat{\mathbf {y}}^\text{S}), \ldots , \mathrm{q}_{|\mathcal {C}|}(\hat{\mathbf {y}}^\text{S})] \) and \( \mathrm{p}(\hat{\mathbf {y}}^\text{T}) = [\mathrm{p}_{1}(\hat{\mathbf {y}}^\text{S}), \ldots , \mathrm{p}_{|C|}(\hat{\mathbf {y}}^\text{T})] \): (4) \( \begin{equation} \mathrm{q}_{k}(\hat{\mathbf {y}}^\text{S}) = \frac{\exp \left(\frac{\hat{\mathbf {y}}^\text{S}_{k}}{\tau } \right)}{\sum _{j \in \mathcal {C}} \exp \left(\frac{\hat{\mathbf {y}}^\text{S}_{j}}{\tau } \right)}, ~~\mathrm{p}_{k}(\hat{\mathbf {y}}^\text{T}) = \frac{\exp \left(\frac{\hat{\mathbf {y}}^\text{T}_{k}}{\tau } \right)}{\sum _{j \in \mathcal {C}} \exp \left(\frac{\hat{\mathbf {y}}^\text{T}_{j}}{\tau } \right)}. \end{equation} \)

Fig. 4.

View Figure

Using the ImageNet dataset, it is empirically shown in Matsubara et al. [91] that all the considered bottleneck-injected student models trained with their teacher models (original models without injected bottlenecks) consistently outperform those trained without the teacher models. This result matches a widely known trend in knowledge distillation reported in Ba and Caruana [3]. However, similar to cross-entropy, the knowledge distillation is still not aware of bottlenecks we introduce to DNN models and may result in significant accuracy loss as suggested by Matsubara et al. [91].

Reconstruction-based Training

As illustrated in Figure 5, Choi et al. [14], Jankowski et al. [57], Sbai et al. [124], and Yao et al. [164] inject AE models into existing DNN models and train the injected components by minimizing the reconstruction error. First, manually an intermediate layer in a DNN model (say its jth layer) is chosen, and the output of the jth layer \( \mathbf {z}_{j} \) is fed to the encoder \( \mathrm{f}_\text{enc} \) whose role is to compress \( \mathbf {z}_{j} \). The encoder’s output \( \mathbf {z}_\text{enc} \) is a compressed representation, i.e., bottleneck, to be transferred to the edge server, and the following decoder \( \mathrm{f}_\text{dec} \) decompresses the compressed representation and returns \( \mathbf {z}_\text{dec} \). As the decoder is designed to reconstruct \( \mathbf {z}_{j} \), its output \( \mathbf {z}_\text{dec} \) should share the same dimensionality with \( \mathbf {z}_{j} \). Then, the injected AEs are trained by minimizing the following reconstruction loss: (5) \( \begin{eqnarray} \mathcal {L}_\text{Recon.}\left(\mathbf {z}_{j} \right) &=& \Vert \mathbf {z}_{j} - \mathrm{f}_\text{dec}(\mathrm{f}_\text{enc}(\mathbf {z}_{j}; \theta _\text{enc}); \mathbf {\theta _\text{dec}}) + \epsilon \Vert _{n}^{m}, \\ &=& \Vert \mathbf {z}_{j} - \mathbf {z}_\text{dec} + \epsilon \Vert _{n}^{m} \nonumber \nonumber , \end{eqnarray} \) where \( \Vert \mathbf {z}\Vert _n^m \) denotes the \( m^\text{th} \) power of the n-norm of \( \mathbf {z} \), and \( \epsilon \) is an optional regularization constant. For example, Choi et al. [14] set \( m = 1 \), \( n = 2, \) and \( \epsilon = 10^{-6} \), and Jankowski et al. [57] use \( m = n = 1 \) and \( \epsilon = 0 \). Inspired by the idea of knowledge distillation [46], Yao et al. [164] also consider additional squared errors between intermediate feature maps from models with and without bottlenecks as additional loss terms like generalized head network distillation [94], described later. While Yao et al. [164] show a high compression rate with small accuracy loss by injecting encoder-decoder architectures into existing DNN models, such strategies [14, 57, 124, 164] increase computational complexity as a result. Suppose the encoder and decoder consist of \( L_\text{enc} \) and \( L_\text{dec} \) layers, respectively; then the total number of layers in the altered DNN model is \( L + L_\text{enc} + L_\text{dec} \).

Fig. 5.

View Figure

Head Network Distillation

The training methods described above are focused on either end-to-end or encoder-decoder training. The first approach often requires hard targets such as one-hot vectors and more training cost, while the latter can focus on the injected components (encoder and decoder) during training, but the additional components (layers) will increase the complexity of the DNN model. To reduce both training cost and model complexity while preserving accuracy, it is proposed in Matsubara et al. [90] to use head network distillation (HND) to distill the head portion of the DNN—which contains a bottleneck—leveraging pretrained DNN models. Figure 6 illustrates this approach.

Fig. 6.

View Figure

The original pretrained DNN (consisting of L layers) is used as a starting point, whose architecture (in the head part) is simplified. As only the teacher’s head portion is altered, the tail portion of the student model is identical to that of the teacher model with respect to architecture and the same pretrained parameters can be maintained. Thus, head network distillation requires only the first layers of the teacher and student models in the training session as the student head model \( \mathrm{f}_\text{head}^\text{S} \) will be trained to mimic the behavior of teacher’s head model \( \mathrm{f}_\text{head}^\text{T} \) given an input \( \mathbf {x} \): (6) \( \begin{equation} \mathcal {L}_\text{HND}(\mathbf {x}) = \big \Vert \mathrm{f}_\text{head}^\text{S}\left(\mathbf {x}; \mathbf {\theta }_\text{head}^\text{S}\right) - \mathrm{f}_\text{head}^\text{T}\left(\mathbf {x}; \mathbf {\theta }_\text{head}^\text{T}\right) \big \Vert ^2 , \end{equation} \) where \( \mathrm{f}_\text{head}^\text{S} \) and \( \mathrm{f}_\text{head}^\text{T} \) are sequences of the first \( L_\text{head}^\text{S} \) and \( L_\text{head}^\text{T} \) layers in student and teacher models (\( L_\text{head}^\text{S} \ll L^\text{S} \), and \( L_\text{head}^\text{T} \ll L \)), respectively.

Experimental results with the ImageNet (ILSVRC 2012) dataset show that given a bottleneck-introduced model, the head network distillation method consistently outperforms cross-entropy-based training [26, 49, 126] and knowledge distillation methods in terms of not only training cost but also accuracy of the trained model. This method is extended in Matsubara and Levorato [94], where the generalized head network distillation technique (GHND) is proposed for complex object detection tasks and models. We note that these tasks require finer feature maps mimicking those at intermediate layers in the original pretrained object detectors. The loss function in this approach is (7) \( \begin{equation} \mathcal {L}_\text{GHND}(\mathbf {x}) = \sum _{j \in \mathcal {J}} \lambda _{j} \cdot \mathcal {L}_{j}\left(\mathbf {x}, \mathrm{f}_{1-L_j^\text{S}}^\text{S}, \mathrm{f}_{1-L_j}^\text{T}\right)\!, \end{equation} \) where j is the loss index, \( \lambda _{j} \) is a scale factor (hyperparameter) associated with loss \( \mathcal {L}_{j} \), and \( \mathrm{f}_{1-L_j^\text{S}}^\text{S} \) and \( \mathrm{f}_{1-L_j^\text{T}}^\text{T} \) indicate the corresponding sequences of the first \( L_j^\text{S} \) and \( L_j^\text{T} \) layers in the student and teacher models (functions of input data \( \mathbf {x} \)), respectively. The total loss, then, is a linear combination of \( |\mathcal {J}| \) weighted losses. Following Equation (7), the previously proposed head network distillation technique [90] can be seen as a special case of generalized head network distillation (GHND). GHND significantly improved the object detection performance in bottleneck-injected R-CNN models on the COCO 2017 dataset while achieving a high compression rate.

5.1 Rationale behind EE

The core idea of EE, first proposed by Teerapittayanon et al. [140], is to circumvent the need to make DNN models smaller by introducing early exits in the DNN, where execution is terminated at the first exit achieving the desired confidence on the input sample. For instance, some samples in test datasets (and in real-world problems) will be easy for a DNN model, but others may not be, depending on the ML models we use. Thus, EE ends the inference process with fewer transforms (layers) for such easy samples so that the overall inference time and computation cost are reduced.

Figure 7 illustrates an example of early classifiers (subbranches) introduced in a DNN model. In this example, the second early classifier has sufficient confidence in its output (class probability is 0.85 out of 1.0) to terminate the inference for the input sample so that the following layers are not executed. Note that all the exits are executed until the desired confidence is reached; that is, the computational complexity up to that point increases. Thus, the classifiers added to the DNN model need to be simple; that is, they need to have fewer layers than the layers after the branches. Otherwise, the overall inference cost will increase on average rather than decrease. Teerapittayanon et al. [141] also apply this idea to mobile-edge-cloud computing systems; the smallest neural model is allocated to the mobile device, and if that model’s confidence for the input is not large enough, the intermediate output is forwarded to the edge server, where inference will continue using a mid-sized neural model with another exit. If the output still does not reach the target confidence, the intermediate layer’s output is forwarded to the cloud, which executes the largest neural model. EE strategies have been widely investigated in the literature, as summarized in Table 4.

Fig. 7.

View Figure

As shown in Tables 1 and 3, most of the studies on SC were focused on computer vision. For EE, we can confirm a good balance between the studies with computer vision and NLP applications as summarized in Table 4, with structural/conceptual differences between the two domains. Moreover, CNN (e.g., AlexNet [64] and ResNet [43]) and Transformer-based models (e.g., BERT [21]) are mostly discussed in the EE studies for computer vision and NLP, respectively. For these reasons, we categorize the EE papers by task domain in Sections 5.2 and 5.3.

5.2 EE for CV Applications

Similar to the SC studies we discussed in Section 4, the research community mainly focused on EE approaches applied to CV tasks.

Design approaches

Wang et al. [150] propose a unified Dual Dynamic Inference that introduces the following features to a DNN model: Input-Adaptive Dynamic Inference (IADI) and Resource-Adaptive Dynamic Inference (RADI). The IADI dynamically determines which sub-networks to be executed for cost-efficient inference, and the RADI leverages the concept of EE to offer “anytime classification.” Using the concept of EE, Lo et al. [87] propose two different methods: (1) authentic operation and (2) dynamic network sizing. The first approach is used to determine whether the model input is transferred to the edge server, and the latter dynamically adjusts the number of layers to be used as an auxiliary neural model deployed on the mobile device for efficient usage of communication channels in EC systems. Neshatpour et al. [102] decompose a DNN’s inference pipeline into multiple stages and introduce EE (termination) points for energy-efficient inference.

Training Approaches

Wang et al. [148] focus on training methods for DNNs with an early exit and observe that prior EE approaches suffered from the burden of manually tuning balancing weights of early-exit losses to find a good tradeoff between computational complexity and overall accuracy. To address this problem, the authors propose a strategy to dynamically adjust the loss weights for the ResNet models they consider. Li et al. [77] and Phuong and Lampert [108] introduce multiple early exits to DNN models and apply knowledge distillation to each of the early exits as students, using their final classifiers as teacher models. Similar to other studies, the DNNs with early exits are designed to finish inference for “easy” samples by early sub-classifiers based on confidence thresholds defined beforehand.

Inference Approaches

Yang et al. [158] leverage EE strategies for multi-scale inputs and propose an approach to classify “easy” samples with smaller neural models. Different from prior studies, their proposed approach scales up the input image (use higher-resolution image as input), depending on the classification difficulty of the sample. Laskaridis et al. [67] design a distributed inference system that employs synergistic device-cloud computation for collaborative inference, including an EE strategy (referred to as progressive inference in their work). Xing et al. [157] apply EE strategies to quality enhancement tasks and propose a resource-efficient blind quality enhancement approach for compressed images. By identifying “easy” samples in the tasks, they dynamically process input samples with/without early exits. Zeng et al. [168] combine EE and SC approaches and propose a framework named Boomerang, which is designed to automate end-to-end DNN inference planning for IoT scenarios; they introduce multiple early exits in AlexNet [64]. Their proposed framework profiles the model to decide its partition (splitting) point.

In addition to introducing and training bottleneck points for object detectors, Matsubara and Levorato [94] introduce a neural filter in an early stage of the head-distilled Keypoint R-CNN model. Similarly to EE frameworks, the filter identifies pictures without objects of interest and triggers termination of the execution before the output of the bottleneck is forwarded. Wołczyk et al. [153] propose Zero Time Waste, a method in which each early exit reuses predictions returned by its predecessors. The method adds direct connections between early exits and combines outputs of the previous early exits like an ensemble model. Through experiments with multiple image classification datasets and model architectures, they demonstrate that their proposed method improves a tradeoff between accuracy and inference time compared to other early-exit methods. Extending the idea of BranchyNet [140], Chiang et al. [12] formulate the early-exit (branch) placement problem. They propose a dynamic programming algorithm to address the problem and discuss the tradeoff between model accuracy and inference time. Pomponi et al. [109] introduce multiple early exits to a classifier and train the entire multi-exit model jointly. Using multiple base models, they discuss various early-exit stopping criteria. Many studies on EE for CV tasks publish their source code to ensure replicability of their work.

5.3 EE for NLP Applications

Interestingly, EE approaches have been widely studied not only in CV tasks—the main application of SC—but also in NLP tasks. Recent studies introduce subbranches (early exits) to transformer-based models such as BERT [21]. While these transformer-based models achieve state-of-the-art performance in NLP tasks, they have an extremely large number of parameters; e.g., BERT [21] has up to 355 million parameters, whereas the largest image classification model used in SC studies (Tables 1 and 3), ResNet-152, has 60 million parameters.

In Elbayad et al. [24] an EE technique for NLP tasks is developed for transformer sequence-to-sequence models [145] in machine translation tasks. The decoder networks in the considered transformer models can be trained by either aligned training or mixed training methods. The former method optimizes all classifiers in the decoder network simultaneously. However, when a different classifier (exit) is chosen for each token (e.g., word) at test time, some of the hidden states from previous time steps may be missed and then the input states to the following decoder network will be misaligned (mismatched). The latter method addresses this issue. In mixed sample training, several paths of random exits are sampled at which the model is assumed to have exited for reducing the mismatch by feeding hidden states from different decoder depths of previous time steps.

For different tasks, Soldaini and Moschitti [132], Xin et al. [156], and Liu et al. [84] propose EE frameworks based on BERT [21] and RoBERTa [85] that share almost the same network architecture. Focused on text ranking, specifically answer sentence selection tasks with question-answering datasets, Soldaini and Moschitti [132] add classification layers to intermediate stages of RoBERTa to build sequential (neural) rerankers [95] inside as early exits and propose the Cascade Transformer models. Focusing on powerful transformer models for industrial scenarios, Liu et al. [84] discuss the effectiveness on 12 (6 English and 6 Chinese) NLP datasets of BERT models when early classifiers are introduced. Similar to the studies by Li et al. [77] and Phuong and Lampert [108], Liu et al. [84] leverage knowledge distillation [46] to train early classifiers, treating the final classifier of the BERT model and their introduced early classifiers as teacher and student classifiers, respectively. Xin et al. [156] target general language understanding evaluation (GLUE) tasks [146] and introduce early exits after each of 12 transformer blocks in BERT and RoBERTa models.

While the Cascade Transformer [132] disregards a fixed portion of candidates (samples) given a query in answer sentence selection tasks, Xin et al. [155] use a score-based EE strategy for a BERT architecture for text ranking tasks. Zhou et al. [172] introduce early classifiers to BERT and ALBERT [66] models and discuss adversarial robustness using the ALBERT models with and without the early exits. Using an adversarial attack method [59], the authors feed perturbed input data (called adversarial examples [65]) to their trained models and show how robust their models are against the adversarial attack, compared to those without early classifiers. Garg and Moschitti [31] propose an approach to filter out questions in answer sentence selection and question-answering tasks. Leveraging the concept of knowledge distillation, they train a question filter model (student), whose input is a query, by mimicking the top-1 candidate score of the answer model (teacher), whose input is a pair of query and the list of the candidate answers. When the trained question filter model finds a query answerable for the answer model, the subsequent inference pipeline will be executed. Otherwise, the question filter model terminates the inference process for the query (i.e., early exit) to save the overall inference cost.

Most of the studies on EE for NLP tasks in Table 4 are published with source code to ensure replicable results. Notably, this application domain enjoys a well-generalized open source framework—Huggingface’s Transformers [154]—which provides state-of-the-art (pretrained) Transformer models, including the BERT, RoBERTa, and ALBERT models used in the above studies.

5.4 Training Methodologies for EE Strategies

To introduce EE strategies, the early classifiers need to be trained in addition to the base models. We can categorize the training methodologies used in EE studies into two main classes: joint training and separate training, illustrated in Figure 8 and described in the next sections.

Fig. 8.

View Figure

Joint Training

Most of the training methods used in existing works belong to this category. Joint training trains all the (early) classifiers in a model simultaneously (left part of Figure 8). Specifically, these studies [24, 67, 87, 109, 132, 140, 141, 148, 150, 155, 157, 158, 168, 172] define a loss function for each of the classifiers and minimize the weighted sum of cross-entropy losses per sample as follows: (8) \( \begin{equation} \mathcal {L}_\text{Joint}([\mathbf {\hat{y}}^{1}, \ldots , \mathbf {\hat{y}}^{N}], c) = \sum _{j=1}^{N} \lambda _{j} \mathcal {L}_\text{CE}(\mathbf {\hat{y}}^{j}, c), \end{equation} \) where \( [\mathbf {\hat{y}}^{1}, \ldots , \mathbf {\hat{y}}^{N}] \) indicates outputs from N (early) classifiers, and the correct label c is shared across all the classifiers in a model. Note that the base model (final classifier) is also counted as one of the N classifiers, and \( N-1 \) early classifiers are introduced to the base model.

Instead, Li et al. [77] and Phuong and Lampert [108] use a knowledge distillation-based loss such as Equation (3) by treating the final classifier (last exit) as a teacher model and all the early classifiers as student models. This approach is based on the assumption that the last classifier will achieve the highest accuracy among all the (early) classifiers in the model, and early classifiers (students) could learn from the last classifier as a teacher model.

Separate Training

A few studies [31, 84, 94, 156] suggest training the early classifiers separately. This approach can be interpreted as a two-stage training paradigm that trains a model in the first stage and then trains the early classifiers introduced to the pretrained model whose parameters are fixed in the second stage (see Figure 8 (right)). For instance, Xin et al. [156] fine-tune a BERT model in the first stage following Devlin et al. [21]. Then, the early classifiers are introduced in the model and trained while all the parameters of the BERT model learned in the first stage are kept frozen. Liu et al. [84] adopt a similar approach, but in the second training stage, knowledge distillation is used to train the early classifiers. Different from SC studies using knowledge distillation, the teacher model is fixed, and only the additional parameters corresponding to the early classifiers are trained. Wołczyk et al. [153] introduce early exits to a pretrained model. Using the cross-entropy loss, they train the introduced early exits.

Evaluation of SC and EE in More Practical Settings

Due to the cross-disciplinary nature of this research area, it is essential to design practical and convincing evaluation settings to demonstrate the effectiveness of proposed approaches. As shown in Tables 3 and 4, the techniques proposed in many of the recent related studies are validated only on small-scale datasets such as MNIST and CIFAR datasets, which leads to some concerns on the input data size in relation to compression. Indeed, Table 2 suggests that the input data size in many of such datasets is relatively small (e.g., smaller than 2 kilobytes per image with a resolution of \( 32 \times 32 \) pixels). The low resolution of the input size may enable conventional EC, where the mobile device fully offloads the computing task by transferring the input data to an edge server. In fact, the transmission of such a small amount of data would require a short time even in settings with limited communication capacity. As a consequence, executing even small head models on a resource-limited mobile device could lead to an overall delay increase.

Based on the above discussion, it becomes apparent that the models and datasets, in addition to the wireless and computing environments, are of paramount importance when assessing the performance of SC and EE schemes. Of particular relevance is the evaluation of accuracy, which is not provided in some of the early studies (e.g., [43, 122, 128]), and the consideration of state-of-the-art models and datasets, which are largely used in the machine learning community. For instance, the use of small models, such as MobileNetV2, ResNet-50, and VGG-16, which are likely overparameterized for simple classification tasks, could lead to wrong conclusions when injecting bottlenecks. Conversely, it was shown in [90] how challenging it is to inject bottlenecks when considering complex vision tasks such as classification on the ImageNet dataset [120].

Optimization of Bottleneck Design and Placement in SC

The study of the architecture and placement of the bottleneck in a DNN model is also of considerable importance. As suggested in [96], important metrics include (1) bottleneck data size (or compression rate), (2) complexity of the head model executed on the mobile device, and (3) resulting model accuracy. As a principle, the smaller the bottleneck representation is, the lower the communication cost between the mobile device and edge server will be. In general, the objective of SC is to generate a bottleneck whose data size is smaller than that of input data such as JPEG file size of input data, which is in turn much smaller than the data size of the input tensor (32-bit floating point), as the communication delay is a key component to reduce overall inference time [90, 91, 94, 158]. Secondly, since mobile devices often have limited computing resources and may have other constraints such as energy consumption due to their battery capacities, SC should aim at minimizing their computational load by making head models as lightweight as possible. For instance, designing a small bottleneck at a very early stage of the DNN model enables a reduction in the computational complexity of the head model [93, 94].

On top of these two criteria, the resulting model accuracy by the bottleneck injection should not be compromised as the introduced bottleneck removes more or less information at the placement compared to the original model. A reasonable lower bound of the model accuracy in SC would be that of widely recognized lightweight models, e.g., MobileNetV2 [122] for ImageNet dataset, considering a local computing system where such lightweight models can be efficiently executed. In general, it is challenging to optimize bottleneck design and placement with respect to all three different metrics, and existing studies empirically design the bottlenecks and determine the placements. Thus, theoretical discussion on bottleneck design and placement should be an interesting research topic for future work.

Dynamic Control of Exits in EE

In most of the recent studies, early exits are used when one of the introduced early classifiers (exits) is confident enough in its prediction. However, users are required to determine a threshold for each of the classifiers beforehand at least for one early classifier in the original model where we introduce the early classifier. For example, if the first classifier’s prediction score is greater than 0.9 in the range of 0.0 and 1.0, then the inference for the input is terminated.

To achieve more efficient inference without significantly sacrificing the accuracy of the original model, the system needs to find a balance between (early) classifiers. As recent studies introduce multiple early exits to a model at different stages, such optimizations are challenging. In addition to manually defining such a threshold for each of the classifiers based on empirical results, a possibly interesting direction is the optimization of the decision-making process, that is, at which (early) classifier we should terminate the inference for a given input, without a set of thresholds defined beforehand based on system characteristics.

Expanding the Application Domain of SC and EE

The application domains of SC and (in minor part) EE remain primarily focused on image classification. This focus may be explained by the size of the input, which makes compression a relevant problem in many settings and the complexity of the models and tasks. However, there are many other unexplored domains from which SC would benefit. Real-time health conditions monitoring via wearable sensors is a notable example of an application where a significant amount of data is transferred from sensors to edge servers such as cellular phones and home hubs. For instance, the detection and monitoring of heart anomalies (e.g., arrhythmia) from ECG [30] require the processing of high-rate samples (e.g., 100 to \( 1,\!000 \) per heart cycle) using high-complexity DNN models [41]. Health monitoring applications pose different challenges compared to CV-based applications. Indeed, in the former, both the computing capacity and the bandwidth available to the system are often smaller compared to the latter scenario, and conceptual advancements are required.

Toward an Information-theoretic Perspective

The key intuition behind the success of SC and EE is similar to what has led to the success of techniques such as model pruning [38, 44, 74, 160] and knowledge distillation [46, 61, 98]: most state-of-the-art DNNs are significantly overparameterized [165, 166]. A possible approach to justify SC and EE can be found in the study of information bottlenecks (IBs), which were introduced in [142] as a compression technique in which a random variable \( \mathbf {X} \) is compressed while preserving relevant information about another random variable \( \mathbf {Y} \). The IB method has been applied in [143] to quantify mutual information between the network layers and derive an information theory limit on DNN efficiency. This has led to attempts at explaining the behavior of deep neural networks with the information bottleneck formalism [123].

Despite these early attempts, a strong connection between this relatively new perspective and the techniques described in this article is still elusive. Some of the approaches and architectures discussed in this article are meaningful attempts to efficiently extract a compressed representation of the input and provide sufficient information toward a certain task early in the network layers. The emerging IB formalism is a promising approach to enable the first moves in the information-theoretic analysis of neural-network-based transformations. We believe that this interpretation could serve as a foundation for an in-depth study of structural properties for both SC and EE.