Optimizing Performance of Federated Person Re-identification: Benchmarking and Analysis

2.1 Person ReID

The objective of person ReID is to retrieve the identity of interest from disjoint camera views. It is an important computer vision task that is widely applied in public security, such as video surveillance [58]. The advances of the DNN have greatly improved the performance of person ReID by learning better feature representations compared to traditional hand-crafted feature operations [29, 31, 46, 52]. Over the years of development, the community has constructed many person ReID datasets [15, 28, 50, 57, 59]. These datasets are collected from various locations with different camera views. The majority of person ReID studies focus on extracting better feature representations by improving the architecture of DNNs [22, 54]. They rely on the assumption that data, collected from different cameras in various locations, can be centralized to a central server. However, centralizing plenty of images of individuals raises potential privacy leakage risks. Different from previous approaches, we proposed FedReID—a new training paradigm for ReID to learn ReID models from decentralized data. FedReID mitigates potential privacy leakage issues, as data is not transferred to a central server.

2.2 Federated Learning

FL is an emerging distributed training technique that trains models with decentralized clients coordinated by a central server [18].

Benchmarks. To facilitate the development of FL, researchers have published several benchmarks and datasets: LEAF [3] is the first benchmark for FL research, containing federated datasets for image classification and natural language processing tasks; Streets [34] is a real-world image dataset collected from street cameras for object detection; and OARF [17] is a benchmark that aims to facilitate a wide range of FL applications, such as trend prediction, recommendation, and sentiment analysis. However, different from these tasks, person ReID is a retrieval task in which no existing benchmark contains related datasets. In this work, we construct a new FL benchmark that simulates real-world scenarios of FedReID.

Algorithm. The most known algorithm for FL is Federated Averaging (FedAvg) [35]. It defines an iterative training process in which clients send trained local models to a server and the server sends back the aggregated global model to clients. The benchmarks mentioned previously adopt FedAvg as the standard algorithm. However, FedAvg requires all clients to have identical models. It is not suitable for FedReID because clients could have varied classifiers. Therefore, we propose an enhanced algorithm, Federated Partial Averaging (FedPav).

Statistical heterogeneity. Statistical heterogeneity—non-IID and unbalanced data—is a major challenge of FL [18, 25]. In traditional distributed training [9, 44], data in multiple nodes of cloud clusters are IID. Data in multiple FL clients, however, could be heterogeneous. To address this challenge, some studies focus on optimizing training in clients [1, 19, 24, 26, 55], whereas recent work [55] requires extra communication by sharing features among clients. However, some studies optimize the aggregation process in the server [47, 48, 61, 62]. In addition, several studies share voluntary or public data between the server and clients [53, 56]. These methods are validated on small datasets [3, 7, 20], and thus may not be directly applicable to the challenging scenario of FedReID. In this work, we introduce three optimization methods targeting the statistical heterogeneity of FedReID via in-depth benchmark analysis.

This work is an extension of our previous conference version [63]. The main improvements are as follows: (1) we introduce a new performance optimization method—CC; (2) we integrate CC with the previously proposed weight adjustment method, achieving the best performance; (3) we conduct more performance evaluations for comparison with the benchmark results and the proposed optimization methods; and (4) we provide more comprehensive descriptions for the proposed optimization methods. Despite that another work [51] also studied FedReID after our conference work [63], it focuses more on adapting to unseen domains, whereas we aim to address the statistical heterogeneity revealed from our benchmark analysis.

3.1 Datasets

We construct the benchmark dataset with nine representative person ReID datasets as shown in Table 1. It contains a total of 224,064 images of 17,991 identities. These datasets are collected at multiple locations (or countries) and published by different organizations at different times. They not only vary in the number of images, identities, and camera views but also differ in image resolution, illumination, and scenes.

The variances in these datasets simulate the statistical heterogeneity in real-world scenarios: the disparity of data volumes represents the unbalanced data problem, and the domain discrepancies among datasets represent the non-IID problem. Unlike centralized training where data is IID, statistical heterogeneity makes training even more challenging.

3.2 Architectures

Figure 1(a), (b) illustrate two architectures for possible implementation scenarios of FedReID: edge-cloud architecture and device-edge-cloud architecture. In both architectures, the cloud represents the central server connecting to multiple edges.

Fig. 1.

View Figure

Edge-cloud architecture. In this architecture, cameras are the edges that directly connect with the server to conduct FL. The server coordinates these cameras to train models with locally collected images. This architecture significantly reduces privacy leakage risks, as the data always stays at the edges. However, deployment of this architecture requires cameras to have enough computation power and storage capability. A real-world application of this architecture would be video surveillance for a community with multiple cameras on different streets.

Device-edge-cloud architecture. This is a three-layer hierarchical architecture. Edge servers are in the middle layer. On the one hand, they construct local training datasets by gathering images from multiple camera views, which is similar to how datasets in the benchmark are collected. On the other hand, edge servers collaboratively perform FL with their local datasets under the coordination of the server. A good illustration of this architecture would be multiple communities collaborating to learn person ReID models, where each community has an edge server collecting data from multiple cameras.

3.3 Algorithm

The standard FL algorithm FedAvg [35] is not suitable for FedReID because it requires identical model structures in all clients. The model structure of the benchmark is ID-discriminative embedding (IDE) [58], which is a common baseline for DNN-based person ReID. This model structure consists of a backbone and a classifier: the backbone is ResNet-50 [13] in our FedReIDBench; the classifier is a linear layer whose dimension depends on the number of identities of clients. Since the number of identities could vary among clients, their classifiers could differ in clients. Hence, we adopt the enhanced algorithm for FedReID: FedPav [63].

FedPav allows models in clients to be only partially identical. For FedReID, FedPav enables clients to use the same backbone but different identity classifiers for FL, as shown in Figure 1(c). The training process is similar to FedAvg except that clients only transfer the identical part of models to the central server for aggregation.

Algorithm 1 summarizes FedPav. We aim to obtain a holistic global model and personalized local models for clients at the end of the training. Each training round \(t\) of FedPav contains four steps: (1) distribution: the central server chooses a fraction (\(K\) out of \(N\)) of clients for current round of training and distributes the global model \(w^t\) to these clients; (2) local training: each client \(k\) initializes the backbone \(w_k^{t}\) using the global model parameters and trains the model with a local dataset for \(E\) local epochs with \(B\) batch size; (3) upload: each client \(k\) uploads the trained backbone \(w_k^{t+1}\) to the server; and (4) aggregation: the server generates a new global model \(w^{t+1}\) by aggregating updates from clients with weighted average. The training stops after iterating these four steps for \(T\) rounds. After training, we use the global model \(w\) to evaluate convergence and generalization, and use local models \(w_k\) to evaluate how well models adapt to local scenarios.

3.4 Performance Evaluation Metrics

We evaluate FedReID in two aspects: accuracy and communication cost.

Accuracy. The cumulative matching characteristics curve and mean Average Precision (mAP) [58] are standard person ReID evaluation metrics. Given an image as a query, person ReID matches it in a gallery of images based on similarity. Cumulative matching characteristics measures the probability that the query identity is in the top-\(k\) most similar matched gallery images. We consider \(k=\lbrace 1, 5, 10\rbrace\) in the benchmark, representing the rank-1 accuracy, rank-5 accuracy, and rank-10 accuracy. In addition, we report the mAP of all queries.

Communication cost. Since FL requires iterative communication between a server and multiple clients, we also consider the communication costs. The total communication cost is \(T \times 2 \times M\), where \(T\) is the number of communication rounds and \(M\) is the transmission message size (model size). \(2\, \times \, M\) is the communication cost of each round, considering both uploading and downloading from clients.

3.5 Reference Implementation

To facilitate ease of use and reproducibility, we open source referenced implementation in GitHub.¹ It includes data preprocessing, proposed algorithm, and optimization methods. We plan to integrate it to EasyFL [60] in the future. In addition, we provide the experimental settings as follows.

Learning rate. The initialized learning rates were different for the identity classifier and the backbone: 0.05 for the identity classifier and 0.005 for the backbone. The learning rate schedulers of both are the same with step size 40 and gamma 0.1. In addition, the learning rate for the server fine-tuning in KD is 0.0005.

Optimizer. We use Stochastic Gradient Decent (SGD) as the optimizer. The optimizer is set with weight decay 5e-4 and momentum 0.9.

FL settings. The default settings of FL algorithms are as follows: batch size \(B = 32\), local epoch \(E = 1\), and total training rounds \(T = 300\).

4.1 Edge-Cloud Architecture

In the edge-cloud architecture, each camera is a client. Since each person ReID dataset contains data from several camera views, we simulate FedReID in this architecture by assigning data of the same camera view to one client. As a dataset is divided into several clients by camera views, we term it the federated-by-camera scenario.

To understand FedReID performance in the federated-by-camera scenario, we compare it with two other settings. The first is the federated-by-identity scenario: we divide one dataset into partitions for multiple clients, where each client includes one partition that contains an equal number of identities. The number of clients equals the number of camera views. The second is centralized training: training with data merged from multiple cameras, which can be considered as the upper bound. For example, the Market-1501 dataset [57] contains six camera views with 751 identities. In the federated-by-identity scenario, we divide it into six clients, where each client includes 125 non-overlapping identities. The centralized training means training with the Market-1501 dataset.

Table 2 presents the comparisons of global models of different settings on two datasets: CUHK03-NP [30] and Market-1501 [57]. Compared with the federated-by-identity scenario or centralized training, the federated-by-camera scenario performs much worse. This indicates that learning from only one camera view is infeasible to obtain a generalized model in person ReID, where the evaluation is based on images from multiple camera views. Hence, even though industrial cameras have enough computation and storage capacity to support edge-cloud architecture, the device-edge-cloud architecture could be more adequate for FedReID because each client learns cross-camera knowledge. All other experiments in the article are conducted based on the device-edge-cloud architecture.

4.2 Device-Edge-Cloud Architecture

In the device-edge-cloud architecture, edge servers collect data from multiple cameras and conduct FedReID with a central server. Since each of the benchmark datasets consists of data from multiple camera views, we simulate this scenario with nine clients—each client contains one unique dataset of the benchmark datasets. In all experiments, we choose nine clients to participate in training.

Under this architecture, we consider two types of models produced from FedReID training: (1) local model: the specialized models trained after \(E\) local epochs in clients before uploading to the server in each training round, and (2) global model: the generalized model obtained in the server by aggregating models uploaded from clients.

To understand the performance of FedReID, we compare global and local models with the other two models: (1) standalone training: the models trained in clients with their own dataset (without participating in FL), and (2) centralized training: the model trained using the combination of all benchmark datasets, simulating conventional person ReID training that centralizes datasets. Centralized training can be treated as the upper bound of FedReID, whereas FedReID is meaningful for a client only when the performance of global or local models is better than standalone training.

4.2.1 Impact of Federated Settings.

We first investigate the performance of FedReID (the global model) using the FedPav algorithm under different federated settings, including batch size \(B\) and local epochs \(E\).

Batch size reflects the trade-off between computation power consumption and model accuracy. With the same local epochs, a larger batch size reduces computation time because the training can better take advantage of the parallelism provided by the client hardware. (Computation is fully utilized as long as \(B\) is large enough.) Figure 2(a) compares the rank-1 accuracy of FedPav using different batch sizes \(B = \lbrace 32, 64, 128\rbrace\), under the setting that local epochs \(E = 1\) and communication rounds \(T = 300\). Smaller batch size generally achieves better performance in most datasets but consumes higher computation.

Fig. 2.

View Figure

Local epochs reflect the trade-off between the communication cost and model accuracy. The total training epochs \(E_{total}\) can be calculated with \(E_{total} = T \times E\), where T is the communication rounds and \(E\) is the number of local epochs. By fixing the total training epochs for a fair comparison, smaller \(E\) means larger communication rounds \(T\), requiring higher communication costs. In addition, we compare the rank-1 accuracy of different numbers of local epochs in Figure 2(b). Despite that \(E = 5\) performs worse than \(E = 10\) in several datasets, smaller numbers of local epochs \(E\) generally result in better performance. The smallest number of local epoch \(E = 1\) achieves much better performance than \(E = 5\) and \(E = 10\) in all datasets, and it requires the highest communication cost, indicating the trade-off between communication costs and model accuracy.

5.1 Client Clustering

To tackle the performance degradation caused by non-IID data in server aggregation of all clients, we propose to aggregate clients with similar data distributions. As discussed in Section 4.2.2, local models outperform the global model in all datasets. The global model is obtained by aggregating local models, so the performance drop mainly sources from the aggregation of clients with diverse data distributions. To tackle this problem, we propose CC to split clients into several groups based on their data distributions and aggregate models within each group in the server.

Figure 3(a) depicts the process of CC with the following steps: (1) we extract features \(f_k\) from one batch data (32 samples) of a public person ReID dataset² using the trained model \(w_k\) from client \(k\); (2) we adopt a clustering algorithm to cluster these features into multiple groups; (3) we aggregate models of clients within each group, obtaining a global model in each group; and (4) we use the global model of each group to update local models of clients within that group for the next training round. In Figure 3(a), we cluster clients into two groups: one group contains clients {1, 4} and another one contains clients {2, 3, 5}, based on their features \(f\). Then, we aggregate \(w_1\) and \(w_4\) to obtain global model \(w_{c1}\), and we aggregate \(w_2\), \(w_3\), and \(w_5\) to obtain \(w_{c2}\). At the start of next training round, we update local models of clients {1, 4} with \(w_{c1}\) and local models of clients {2, 3, 5} with \(w_{c2}\). CC obtains multiple global models after training, so we focus on evaluating personalized local models \(w_k\) of each client \(k\).

Fig. 3.

View Figure

In this way, we use the features as a proxy to measure the similarity of data distributions among clients. The intuition behind CC is that the clients clustered into the same group share more similar data distributions. The choice of the clustering algorithm is important for the overall performance of this method. We utilize a hierarchical clustering algorithm, FINCH [39], to cluster clients based on similarities of features extracted from their models. Regarding each client as a cluster at the start, we group the clients that are first neighbors; two clients are first neighbors if their features have the shortest distance (cosine similarity) or they share the same first neighbor. FINCH merges first neighbors in each clustering step. In our scenario, since nine clients would be merged into one cluster after two to three clustering steps, we only cluster for one step per communication round. As a result, the server would have two to three clusters, where each cluster contains two to seven clients. FINCH is able to deliver good clustering results without prior knowledge of the targeted number of clusters.

5.2 Knowledge Distillation

Besides CC, we adopt KD to elevate performance and improve the convergence of FedReID. Since local models outperform the global model, this suggests that local models contain more knowledge than the global model—simple server aggregation could not effectively aggregate knowledge from local models. KD is a method proposed by Hinton et al. [14] to transfer knowledge from a teacher model to a student model, where the teacher model contains more knowledge than the student model. We adopt KD to better transfer knowledge from local models to the global model, regarding clients as teachers and the server as the student.

After clients finish local training and upload models, we apply KD with a public shared dataset \(\mathcal {D}_{shared}\) in the server. Figure 3(b) illustrates the additional steps required from KD. In the first step, the server uses each trained model \(w_k\) of client \(k\) to generate soft labels³ \(\ell _k\) using samples of \(\mathcal {D}_{shared}\). These soft labels represent the knowledge of clients’ models. In the second step, apart from model aggregation, the server aggregates these soft labels with \(\ell = \frac{1}{K} \sum _{k \in S_t} \ell _k\). In the third step, the server fine-tunes the global model with \(\mathcal {D}_{shared}\) and corresponding labels \(\ell\) to learn the distilled knowledge.

5.3 Weight Adjustment

In addition to tackling the performance degradation caused by non-IID data, we propose to dynamically update the weights for aggregation to curb the adverse effect of unbalanced data. As discussed in Section 4.2.2, the weights of server aggregation are inappropriate. The formula for server aggregation [35, 63] is \(w^{t+1} = \sum _{k \in S_t} \frac{n_k}{n} w^{t+1}_k\), where \(n\) is the total data volume and \(n_k\) is the data volume of client \(k\). The weights of local models depend on the data volume of clients—larger datasets lead to larger weights. Since data volumes have large discrepancies among datasets, large datasets dominate in the server aggregation. For example, the weight of the largest dataset (MSMT17 [50]) is around 40%, whereas the weight of smallest dataset (iLIDS-VID [49]) is only 0.3%. Models from smaller datasets are almost negligible in aggregation. Such unbalanced data volume hampers clients with large datasets to effectively learn from others. Hence, we introduce a novel weight adjustment method to obtain more suitable weights for weighted average in aggregation.

Cosine distance weight. We introduce cosine distance weight (CDW) to substitute the weight of data volumes. CDW adjust the weights for aggregation dynamically in each round, based on how well models are trained in clients. It is measured by changes in features extracted from models before and after training. Such changes are calculated by cosine distance. In particular, in each training round, client \(k\) downloads and trains on the global model \(w^t_k\) from the server to obtain a new local model \(w^{t+1}_k\). Figure 3(c) demonstrates our method to calculate the new weight with \(w^t_k\) and \(w^{t+1}_k\), with the following steps: (1) client \(k\) extracts logits \(g^t_k\) with a random batch data \(\mathcal {D}_{batch}\) using \((w^t_k, v^t_k)\)⁴; (2) client \(k\) obtains a new local model \((w^{t+1}_k, v^{t+1}_k)\) after local training; (3) client \(k\) extracts features \(g^{t+1}_k\) with \(\mathcal {D}_{batch}\) using \((w^{t+1}_k, v^{t+1}_k)\); and (4) we calculate the cosine distance of these two logits \(g^{t}_k\) and \(g^{t+1}_k\), with following formula: (1) \(\begin{equation} d^{t+1}_k = 1 - \frac{g^{t}_k \cdot g^{t+1}_k}{\left\Vert g^{t}_k \right\Vert \left\Vert g^{t+1}_k \right\Vert }, \end{equation}\) where the cosine distance \(d^{t+1}_k\) of each client \(k\) is pushed to the server. The server uses the following formula to obtain the new weight: (2) \(\begin{equation} p^{t+1}_k = \frac{d^{t+1}_k}{\sum _{k \in S_t} d^{t+1}_k}, \end{equation}\) where the server uses \(p^{t+1}_k\) to replace \(\frac{n_k}{n}\) as the new weight for aggregation. The intuition of CDW is that clients whose local trainings are more effective should contribute more to the aggregation. The cosine distance \(d_k\) measures the scale of changes in local training that updates model \(w^t_k\) to \(w^{t+1}_k\).

5.4 Combinations of Optimization Method

We can achieve even better performance by combinations of these three optimization methods: CC, KD, and CDW. We consider only combining CDW and CC, and CDW and KD.

It is not desirable to combine CC and KD because they both enhance the server aggregation. On the one hand, KD only fine-tunes a single global model, whereas CC contains multiple global models. On the other hand, both methods address the non-IID problem: KD aims to further improve the global model, whereas CC tends to elevate the performance of local models. Hence, we do not consider the combination of these two methods.

Since CDW tackles unbalanced data volume, either the combination of CDW and CC or the combination of CDW and KD addresses statistical heterogeneity with non-IID and unbalanced data problems. To combine CDW with CC or KD, we just need to replace the original weights in the server aggregation process with the new weights. As CC has no single global model, combining it with CDW aims to achieve better local models; as KD further fine-tunes the global model, combining it with CDW aims to achieve a better global model. We summarize these two combinations in Algorithms 2 and 3.

5.5 Evaluation

We present the empirical evaluation of these performance optimization approaches compared with the benchmark and standalone training. By default, we conduct these experiments with batch size \(B = 32\) and local epoch \(E = 1\). For both CC and KD, we adopt an additional unlabeled dataset—CUHK02 [27]. This dataset is regarded as a public dataset that is shareable among clients and the server. The CUHK02 dataset is an extension of the CUHK01 dataset. It includes 7,264 images of 1,816 identities collected from six camera views.

We first evaluate the effectiveness of KD and the combination of CDW and KD by monitoring performance changes of global models as training proceeds. Figure 4 shows the performance changes (either rank-1 accuracy or mAP) of KD, the combination of CDW and KD, and the benchmark results on eight datasets. Compared with the benchmark results, training with KD achieves much better convergence; KD can also lead to higher performance, especially when datasets in clients share similar data distributions with the public shared dataset. For example, we use the CUHK02 dataset as the shared dataset, so the accuracy of the global models on the CUHK03-NP and CUHK01 datasets is better than the benchmark results. Moreover, training with the combination of KD and CDW achieves outstanding performance on almost all datasets—better than the benchmark results or training with KD. These results indicate that the combination of KD and CDW is able to obtain the best generalized global model that is transferable to other scenarios.

Fig. 4.

View Figure

Next, we evaluate the effectiveness of CC, CDW, and the combination of these two methods by comparing the performance of their local models. Table 4 shows the increase in rank-1 accuracy of several methods when compared with standalone training on nine datasets. Although FedNova [48] and FedProx [26] slightly improve the performance of the smallest dataset (iLIDS-VID), they are incapable to elevate the performance of large datasets, like our benchmark method. We further analyze the results in three folds. First, CC effectively mitigates the drawback of the benchmark, improving the performance on larger datasets such as MSMT17 [50]. This is because the dominance of larger datasets over smaller datasets is reduced as they are clustered into different groups in aggregation. Most of the time, CC creates two clusters: the first one contains clients with PRID2011, CUHK03-NP, VIPeR, 3DPeS, and iLIDS-VID datasets; the second one contains clients with MSMT17, DukeMTMC-reID, Market-1501, and CUHK01 datasets. Second, CDW outperforms the standalone training on all datasets. This indicates that CDW effectively addresses the unbalanced data problem such that all clients are beneficial in FL. Third, the combination of CDW and CC further elevates the performance in most datasets. Although such combination produces slight decreases on smaller datasets compared with CDW, it significantly further improves the performance of larger datasets. It increases the motivation of clients with larger datasets to participate in FL.

Last, we demonstrate the generalization ability of our methods by comparing existing methods on the CAVIAR4REID [6] and GRID [33] datasets. Specifically, we compare with the unsupervised cross-domain fine-tuning methods DSTML [16] and UMDL [38]; the unsupervised generalization methods CrossGrad [40], MLDS [23], SSDAL [43], and DIMN [42]; and recent work [51]. For evaluation on CAVIAR4REID, we follow the work of Liu et al. [32] and Peng et al. [38] to randomly select 36 identities that appeared on two camera views. The GRID dataset contains 250 identities from two camera views. For both datasets, we use images of one camera view as the query and another one as the gallery. Table 5 shows that our proposed FedReID with optimizations (CC & CDW and KD & CDW) outperforms all existing methods on rank-1 accuracy on both datasets; KD & CDW achieves especially good performance. Note that we do not fine-tune trained models on these two evaluation datasets. These results further illustrate the significance of our methods.