Federated Multi-view Learning for Private Medical Data Integration and Analysis

1 INTRODUCTION

With the recent advance of technology, medical treatment has gradually become digitized, and a large and heterogeneous amount of medical data (e.g., electronic health records (EHRs) [22], claims, laboratory tests, imaging, and genomics) has been accumulated in medical institutions. The availability of these data offers many potential benefits for healthcare. It not only facilitates sharing information in care-related activities but also reduces medical errors and service time. Moreover, data sharing in the multi-institutional and large-scale collaboration context can effectively improve the generalizability of research, accelerate progress, enhance collaborations among institutions, and lead to new discoveries from data pooled from multiple sources [42]. For these reasons, substantial efforts have been made in the past to improve the medical information systems enabling the collection and processing of huge amounts of data to work toward a multi-institutional collaborative setting. For example, District Health Information Software 2 (DHIS2) [13] is an open-source, web-based Health Management Information System (HMIS) widely used by countries for national-level aggregation of medical data. With the broad adoption of HMISs, security and privacy become critical [47], as medical data are highly sensitive to the patients.

In the past two decades, great efforts have been made to facilitate the availability and integrity of medical data while protecting confidentiality and privacy in medical information systems [2, 3]. Most of the research is centered around data de-identification [16, 17, 50, 53] and data anonymization [14, 39, 43, 55], which removes the identifiable information from the published medical data to prevent an adversary from reasoning about the patients’ privacy. However, as pointed out in Reference [38], published medical data is not the only source that the adversaries can count on: with a large amount of information that people voluntarily share on the Web, sophisticated attacks that join disparate information pieces from multiple sources against medical data privacy become practical. As a result, many medical institutions are unwilling to share their data, as sharing may cause sensitive information to be leaked to researchers, other institutions, and unauthorized users. Even in multi-institutional collaborative research, they are expected to perform integrated analysis without leaving their medical data outside institutions. Therefore, security and privacy become an obstacle and challenge for data integration in medical research, especially when the data are required to be shared for secondary use.

However, machine learning methods [1, 30] often require a huge amount of data for better performance, and the current circumstances put deploying or developing medical AI applications in an extremely difficult situation. To address the data limitation and isolation issues, great progress has been made in the development of secure machine learning frameworks in recent years. A popular approach is the use of federated learning (FL) to support collaborative and distributed learning processes, which enables a large number of resource-limited client nodes to cooperatively train a model without data sharing [44]. For example, FL can be used to process EHR data distributed in multiple hospitals by sharing the local model instead of the patients’ data to prevent the raw data leakage [6, 7, 27, 37].

In the literature, many studies have explored the variants of FL schemes to support complicated tasks in real life. For example, Yang et al. [58] introduced a comprehensive federated learning framework that contains vertical federated learning (VFL) and horizontal federated learning (HFL) based on different local data availability. Specifically, VFL deals with the case that the datasets share the same sample ID space but their feature spaces are different, while HFL deals with the case that the datasets share the same feature space but are different in sample ID spaces. However, the previous works mainly focus on the difficulties in the configuration of FL because of the data distributions but less consider the data complexity. In medical applications, many datasets are complex, heterogeneous and often collected from different instruments and/or measures, known as “multi-view” data [56]. For example, EHRs contain different types of patient-level variables, such as demographics, diagnoses, problem lists, medications, vital signs, and laboratory data. The mobile keyboard data are composed of various sensor data and keystroke records. It has been shown that multi-view learning can get better performance than the single view counterpart, especially when the strengths of one view complement the weaknesses of the other. Nevertheless, federated learning from multi-view data is still in its infancy, as most current solutions can only handle the single-view data. Moreover, existing works are only designed for a static setting without considering the temporal information of the data (e.g., EHRs).

Motivated by the aforementioned problems, this work focuses on multi-view learning tasks and associated problems of federated learning. Specifically, we first present a basic multi-view learning method to fuse multi-view data. Then, based on two types of local data availability, i.e., horizontal and vertical, we adapt the basic multi-view learning method to develop two federated multi-view algorithms: V-FedMV and H-FedMV. Both V-FedMV and H-FedMV approaches can perform the same tasks, but for different multi-view data distributions. In the V-FedMV approach, each client has the same participants, but has only a single-view data for each participant. In the H-FedMV approach, every client has multi-view data equal to a subset of the overall data, but each of them has very limited participants. Third, we investigate how multi-view sequential data can be arranged in the setting of federated learning and present a sequential modelling strategy (S-FedMV) to consider the temporal information of the multi-view data. These methods together provide flexible and effective tools to support multi-institutional collaborations for multi-view data mining while solving the privacy and security challenges for data sharing in medical research.

Our contributions can be summarized as follows:

• To the best of our knowledge, we are the first group that aims to propose a systematic solution for federated multi-view learning.

• We have considered two types of distributed multi-view data and proposed V-FedMV and H-FedMV to deal with these two cases separately.

• This is the first work to consider multi-view sequential data in the federated learning setting, and S-FedMV method is developed in this regard.

• Based on the experimental results, all three proposed methods (V-FedMV, H-FedMV, and S-FedMV) can make full use of multi-view data and are more effective than local training.

The rest of this article is organized as follows. Section 2 describes the problem definition. Section 3 introduces the multi-view learning method and optimization. The V-FedMV and H-FedMV methods are presented in Sections 4 and 5, respectively. Then in Section 6, we show how to adapt the model to deal with sequential data. In Section 7, we briefly discuss the privacy issue. The dataset, experiments and results are presented in Section 8. Section 9 discusses the related work, followed by the conclusion in Section 10.

2 PROBLEM DEFINITION

In this section, we state the problem of federated multi-view learning task. Without loss of generality, we consider the classification problem. Assume that we have a distributed multi-view dataset available with N instances from K views: let \( \mathbf {X}_k\in \mathbb {R}^{N\times d_k} \) denote the data matrix in the kth view, where the ith row \( \mathbf {x}_{ik} \in \mathbb {R}^{d_{k}} \) is the feature descriptor of the ith instance in the kth view. Suppose these N instances are sampled from C classes and denote \( \mathbf {Y}= [\mathbf {y}_1, \ldots , \mathbf {y}_N]^T \in \lbrace 0, 1\rbrace ^{N \times C} \), where \( \mathbf {y}_i \in \lbrace 0, 1\rbrace ^{1 \times C} \) is the one-hot label indicator vector of the ith instance.

The key challenge to the adoption of distributed data is the security and privacy of highly sensitive data. For example, in many situations, medical data cannot leave the institutions, which limits the usefulness of these data to perform analytics in the data aggregation process. To solve this issue, federated learning has emerged as a promising technique for distributing machine learning (ML) model training, which performs local model training, and upload model parameters for global aggregation; thus, it enables collaborative model training while preserving each participant’s privacy, which is particularly beneficial to the medical field.

Based on the local data availability, we suppose that there are two types of federated multi-view learning: horizontal and vertical. Figure 1 shows an example of horizontal and vertical distributions of data. Assume that there are three hospitals, and they want to build a predictive model to diagnose a specific disease in a patient. In some cases, each of hospitals has only a single view-point for each patient (as illustrated in Figure 1(a)), and they are not willing to share the data due to confidentiality of the data. As a result, we cannot directly combine the data from different views together. In some other cases, each hospital has multiple devices and can collect data from different views by their own (as illustrated in Figure 1(b)). However, each of them has only a limited number of patients, and could not be used to develop robust and accurate models for real-world application. Our goal is to leverage all available data without sharing data between institutions for model learning in these respective scenarios, by distributing the model-training to the data-owners and aggregating their results. The problems are formally defined as follows:

Fig. 1.

View Figure

• Vertical Federated Multi-view Learning (V-FedMV): This scenario assumes that K clients are sharing the same ID space, but each of them has different single-view data. The data \( \mathbf {X}=\lbrace \mathbf {X}_1,\mathbf {X}_2,\cdots ,\mathbf {X}_K\rbrace \) is distributed over K clients, which is denoted by \( \lbrace L_1,L_2,\ldots ,L_K\rbrace \); in other words, each client \( L_k \) owns a specific single-view data \( \mathbf {X}_k \). Sharing the same ID space means the ith rows across different \( \mathbf {X}_k \) represent the same instance. Our goal is to generate more accurate and robust classification results, by collectively training multiple single-view models through federated learning, to exploit vertical-level view information across multiple decentralized sites holding local private data samples without exchanging.

• Horizontal Federated Multi-view Learning (H-FedMV): This scenario assumes that there are M clients, each of them owns k-view data but shares different ID spaces, denoted by \( \lbrace L_l\rbrace ^M_{l=1} \). Let \( \mathbf {X}_k^l \in \mathbb {R}^{N_l \times d_k} \) denote the kth single-view data on the lth client, where \( N_l \) represents the number of samples on lth client. Then the data owned by \( L_l \) is defined as \( \mathbf {X}^l=\lbrace \mathbf {X}_1^l,\cdots ,\mathbf {X}^l_K\rbrace \) and can be seen as a subset of the overall data. Our goal is to incorporate all samples in the model training to make the result more promising, by collaboratively training multiple multi-view models with the benefit of the federated learning framework.

Table 1 provides the list of important notations and their definitions that are used in this study.

3 MULTI-VIEW LEARNING

Multi-view learning, also known as data fusion, aims to integrate multiple data information from different views to improve the performance of learning tasks [23, 60]. To deal with the federated multi-view learning problems in the vertical and horizontal situations, we start with a basic multi-view learning method to fuse multi-view data in this section, and then show how the formulas are modified to accommodate vertical and horizontal cases in the following two sections. Our approach is inspired by Reference [18], while the work proposed in Reference [18] aimed at learning multi-class label-sharing model in a privacy-preserving way, and the goal of this study is to process the distributed and heterogeneous multi-view data in a privacy-preserving manner.

3.2 Optimization

We illustrate how to solve Equation (1) in the training phase and how to solve Equation (2) in the testing phase, respectively. Since the objective functions are non-convex and potentially non-smooth, we iteratively update the parameters one by one while fixing the other parameters [18]. The overall algorithm is summarized in Algorithm 1.

3.2.1 Training Phase.

In the training phase, all parameters are iteratively updated according to the following three steps until Equation (1) converges or reaches a maximum number of iterations.

Step 1: update \( \mathbf {W}_k \). When \( \mathbf {Z}_k \) and \( \mathbf {Z} \) are fixed, the optimization problem of minimizing Equation (1) over \( \mathbf {W}_k \) can be written as (3) \( \begin{equation} \underset{\mathbf {W}_k}{min}\sum _{k=1}^K\Vert \mathbf {X}_k\mathbf {W}_k-\mathbf {Z}_k\Vert _F^2 +\beta _k\Vert \mathbf {W}_k\Vert _{2,1}. \end{equation} \)

Following Reference [26], Equation (3) can be written as (4) \( \begin{equation} \underset{\mathbf {W}_k,\,\mathbf {A}_k}{min}\Vert \mathbf {X}_k\mathbf {W}_k-\mathbf {Z}_k\Vert _F^2 +\beta _k\mathrm{Tr}\left(\mathbf {W}_k^\mathrm{T}\mathbf {A}_k\mathbf {W}_k\right)\!, \end{equation} \) where \( \mathbf {A}_k\in \mathbb {R}^{d_k\times d_k} \) is a diagonal matrix with diagonal elements and can be written as (5) \( \begin{equation} \mathbf {A}_k^{(i,j)} = \left\lbrace \!\!\! \begin{array}{cc} 1/\left[2\left(\left\Vert \mathbf {W}_k^{(i)}\right\Vert _2+\epsilon \right)\right] &i=j, \\ [3pt] 0 & i\ne j. \end{array} \right. \end{equation} \)

Here, \( \Vert \mathbf {W}_k^{(i)}\Vert _2 \) denotes the \( \mathcal {l}_{2} \) norm of \( \mathbf {W}_k^{(i)} \) that is the vector at the ith row of \( \mathbf {W}_k \), and \( \epsilon \) denotes a small constant to avoid overflow.

Then, when \( \mathbf {A}_k \) is fixed, \( \mathbf {W}_k \) can be updated by (6) \( \begin{equation} \mathbf {W}_k = \left(\mathbf {X}_k^\mathrm{T}\mathbf {X}_k + \beta _k\mathbf {A}_k\right)^{-1}\mathbf {X}_k^\mathrm{T}\mathbf {Z}_k. \end{equation} \)

Specifically, when \( \mathbf {Z}_k \) and \( \mathbf {Z} \) are fixed, we can repeat the following operation to get \( \mathbf {W}_k \) until the value of Equation (3) converges: fix \( \mathbf {W}_k \) to update \( \mathbf {A}_k \) through Equation (5) and then fix \( \mathbf {A}_k \) to update \( \mathbf {W}_k \) through Equation (6).

Step 2: update \( \mathbf {Z}_k \). When \( \mathbf {Z} \) and \( \mathbf {W}_k \) are fixed, Equation (1) becomes (7) \( \begin{equation} \underset{\mathbf {Z}_k}{min}\sum _{k=1}^K\Vert \mathbf {X}_k\mathbf {W}_k-\mathbf {Z}_k\Vert _F^2 +\zeta _k\Vert \mathbf {Z}_k-\mathbf {Z}\Vert _F^2. \end{equation} \)

\( \mathbf {Z}_k \) can be updated directly by (8) \( \begin{equation} \mathbf {Z}_k = (\mathbf {X}_k\mathbf {W}_k + \zeta _k\mathbf {Z})/(1+\zeta _k). \end{equation} \)

Step 3: update \( \mathbf {Z} \). When \( \mathbf {Z}_k \) and \( \mathbf {W}_k \) are fixed, Equation (1) becomes (9) \( \begin{equation} \underset{\mathbf {Z}}{min}\sum _{k=1}^K \zeta _k\Vert \mathbf {Z}_k-\mathbf {Z}\Vert _F^2+\eta \Vert \mathbf {Z}-\mathbf {Y}\Vert _F^2. \end{equation} \)

Then \( \mathbf {Z} \) can be updated directly by (10) \( \begin{equation} \mathbf {Z} = \left(\sum _{k=1}^K\zeta _k\mathbf {Z}_k + \eta \mathbf {Y}\right)/\left(\sum _{k=1}^K\zeta _k + \eta \right)\!. \end{equation} \)

3.2.2 Testing Phase.

First, initialize \( \mathbf {Z}^{test}_k \) as \( \mathbf {Z}^{test}_k = \mathbf {X}^{test}_k\mathbf {W}_k \). When \( \mathbf {Z}^{test}_k \) is fixed, Equation (2) can be written as (11) \( \begin{equation} \underset{\mathbf {Z}^{test}}{min}\sum _{k=1}^K\zeta _k\Vert \mathbf {Z}^{test}_k-\mathbf {Z}^{test}\Vert _F^2. \end{equation} \)

Then \( \mathbf {Z}^{test} \) can be updated directly by (12) \( \begin{equation} \mathbf {Z}^{test} = \sum _{k=1}^K\zeta _k\mathbf {Z}^{test}_k/\sum _{k=1}^K\zeta _k. \end{equation} \)

When \( \mathbf {Z}^{test} \) is fixed, then \( \mathbf {Z}^{test}_k \) can be updated directly by (13) \( \begin{equation} \mathbf {Z}^{test}_k = (\mathbf {X}^{test}_k\mathbf {W}_k+\zeta _k\mathbf {Z}^{test})/(1+\zeta _k). \end{equation} \)

4 VERTICAL FEDERATED MULTI-VIEW LEARNING

In this section, we elaborate on how proper design of the above multi-view learning mechanism can yield guaranteed vertical federated multi-view learning gains, which consequently not only allows us to connect multi-view relations over a global server for remote collaboration but also to set up an environment across multiple decentralized data to distribute the model-training to the data-owners, without sharing data between institutions.

The overall algorithm of V-FedMV is summarized in Algorithm 3. Briefly, the main procedures of V-FedMV can be described as follows: We first establish a server with multiple distributed clients, then the computations involving the original sensitive data will be executed on clients, while some other computations related to insensitive data of all clients will be executed on the server. For example, Equations (5), (6), (8), and (13) can be computed on each client \( L_k \), while Equations (10) and (12) can be computed on the server.

Figure 2 provides an overview of V-FedMV for vertical accumulation of different views of data for multi-view classification, where the cycle at the bottom of the figure shows the workflow of this system. Assume that there are a server and K hospitals with private single-view data on each of them. First, the server sends each hospital the \( \mathbf {Z} \), which would be used to update the local model with local data on each hospital. Then the \( \mathbf {Z}_k \) would be obtained and sent to the server to be aggregated, resulting in a new \( \mathbf {Z} \).

Fig. 2.

View Figure

5 HORIZONTAL FEDERATED MULTI-VIEW LEARNING

In this section, we illustrate how multi-view learning mechanism can be used for horizontal federated multi-view learning, which consequently allows us to leverage all available multi-view samples from a collection of decentralized local data over a central global server for remote collaboration, but without delivering any raw data to the global server. Figure 3 provides an overview of H-FedMV for horizontal accumulation of samples for multi-view classification. Compared to the vertical case, this system has data available from multiple views. Thus, the hospitals and server exchange the transformation matrices instead of \( \mathbf {Z} \) and \( \mathbf {Z}_k \).

Fig. 3.

View Figure

Before proceeding, we first redefine the notation, since the setting of H-FedMV is different from V-FedMV. Suppose there are M clients, each of which owns multi-view data but has different patients, denoted by \( \lbrace L_l\rbrace _{l = 1}^M \). Let \( \mathbf {X}^l_k \in \mathbb {R}^{N_l \times d_k} \) denote the kth single-view data on the lth client, where \( N_l \) represents the number of samples on lth client. Then the data owned by \( L_l \) is defined as \( \mathbf {X}^l=\lbrace \mathbf {X}^l_1,\ldots ,\mathbf {X}^l_K\rbrace \). The goal of H-FedMV is to use more samples from different institutes or hospitals without data sharing for training models to increase the spectrum of patients incorporated in learning, thereby giving better predictive performance than the individual local averages and preserving data privacy.

Based on the multi-view learning mechanism, we can easily generalize it to the case of horizontal federated learning, in which we perform the multi-view learning algorithm on each client separately. To indicate the data is owned by the lth client, we add the superscript l to each equation. Accordingly, Equations (5), (6), (8), (10), (12), and (13) can be turned into the following forms: (14) \( \begin{equation} \mathbf {A}_k^{l (i,j)} = \left\lbrace \!\!\! \begin{array}{cc} \frac{1}{2\left(\left\Vert \mathbf {W}_k^{l (i)}\right\Vert _2+\epsilon \right)} &i=j, \\ 0 &i\ne j. \end{array} \right. \end{equation} \) (15) \( \begin{equation} \mathbf {W}_k^l = \left[\left(\mathbf {X}_k^l\right)^\mathrm{T}\mathbf {X}_k^l + \beta _k^l\mathbf {A}_k^{l}\right]^{-1}\left(\mathbf {X}_k^l\right)^\mathrm{T}\mathbf {Z}_k^l, \end{equation} \) (16) \( \begin{equation} \mathbf {Z}_k^l = \left(\mathbf {X}_k^l\mathbf {W}_k^l + \zeta _k^l\mathbf {Z}^l\right)/(1+\zeta _k^l), \end{equation} \) (17) \( \begin{equation} \mathbf {Z}^l = \left(\sum _{k=1}^K\zeta _k^l\mathbf {Z}_k^l + \eta ^l\mathbf {Y}^l\right)/\left(\sum _{k=1}^K\zeta _k^l + \eta ^l\right)\!, \end{equation} \) (18) \( \begin{equation} \mathbf {Z}^{l,test} = \sum _{k=1}^K\zeta _k^l\mathbf {Z}^{l,test}_k/\sum _{k=1}^K\zeta _k^l, \end{equation} \) (19) \( \begin{equation} \mathbf {Z}^{l,test}_k = \left(\mathbf {X}^{l,test}_k\mathbf {W}_k+\zeta _k^l\mathbf {Z}^{l,test}\right)/\left(1+\zeta _k^l\right)\!. \end{equation} \)

The overall algorithm of H-FedMV is summarized in Algorithm 3. Briefly, the main procedures can be described as follows: first, all clients apply the algorithm of the multi-view learning on their own data and devices, then each of them can get \( \mathbf {W}_k^l \), which are sent to the global server to compute the weighted average value. After the computation on the server, it sends the output weighted average transformation matrix \( \mathbf {W}_k \) back to the local client \( L_l \) for distributed model training. This process will be repeated several times until convergence is reached or some stopping criteria are met. At last, the \( \mathbf {W}_k \) of all K views are obtained on the server and then the server will send them to clients, which can be used to predict the testing data of each client.

6 FEDERATED MULTI-VIEW SEQUENTIAL LEARNING

In medical practice, sequential or longitudinal data is very common, such as clinical research and epidemiological studies. However, V-FedMV and H-FedMV cannot be directly applied to sequential data for considering the temporal information of the multi-view data. We further proposed a federated multi-view sequential learning method to adapt our model to deal with multi-view sequential data in a federated environment.

In the vertical setting, each client contains a single-view data, and can be independently executed (such as using GRU) on their own data to get the feature embedding matrix from sequential data and then apply the proposed V-FedMV method, which will not expose the raw data in the whole process. However, in the horizontal setting, each client owns multi-view data partially and they cannot process their own data independently, because the trained GRU models for each view are inconsistent locally. Hence, how to do federated multi-view learning in the horizontal setting when the raw data is sequential is a problem. Here, we develop a sequential version of H-FedMV to address this problem, named S-FedMV.

Figure 4 provides an overview of S-FedMV for horizontal accumulation of different samples of multi-view sequential data for feature extraction. As shown in Figure 4, there are M hospitals containing K-view sequential data. Since the data of each view are separately processed, it will finally result in K models on the server, which can be used to extract features for H-FedMV. Briefly, the idea is based on the Federated Averaging (FedAvg), following a server-client setup with two repeated stages: (i) the clients train their models locally on their data, and (ii) the server collects and aggregates the models to obtain a global model by weighted averaging. For each client, we assign the aggregation weight as the ratio of data samples on each client to the total number of samples, namely, \( N_l/N \). It is appropriate to process the data of each view separately in the case of distributed horizontal multi-view data, because there exists inconsistency among the models of different views. FedAvg is flexible to the model and the optimizer used for training, here we use the bidirectional GRU as the model and RMSProp as the optimizer in the experiments. For simplicity, in every round we choose all clients to participate in.

Fig. 4.

View Figure

The overall algorithm is summarized in Algorithm 1, and the notations used in the proposed model are as below: As before, we use \( \lbrace L_l\rbrace \) to denote M clients, \( l=1,\ldots ,M \). Let \( \lbrace D^l_k\rbrace \) denote the sequential data on lth client in the kth view, \( \mathbf {w}_t^k \) and \( \mathbf {w}_t^{k,l} \) denote the parameters of kth view model on server and on the lth client in the t round. \( \mathcal {l}^k(\mathbf {w};b) \) denotes the loss of the model of kth view with parameters \( \mathbf {w} \), and b denotes a batch.

7 DISCUSSION

In V-FedMV, both \( \mathbf {W}_k \) and \( \mathbf {Z}_k \) are updated on local devices while \( \mathbf {Z} \) is updated on the server. \( \zeta _k \) and \( \mathbf {Z}_k \) are sent to the server from local devices when the server updates \( \mathbf {Z} \), which would not leak the raw data, because \( \mathbf {X}_k \) is preserved at local devices throughout the process. Besides, in some cases, the server may not be set up by these clients but by another specialized organization that would not provide the raw data but only do the computation, and in this case, although \( \mathbf {Y} \) is also needed when server updates \( \mathbf {Z} \), it will not cause leakage, because \( \mathbf {Y} \) is only a one-hot label matrix that would not be useful without extra information.

In H-FedMV and S-FedMV, each client sends the parameters of its own model to the server in each round, and the server averages it and returns the result. Each client then continues to update locally with the parameters returned by the server. In the whole process, the server only contacts the parameters, not the raw data, and the local devices can only contact the updated parameters from the server, not the raw data from other clients, so this ensures privacy.

8 EXPERIMENTAL EVALUATION

In this section, we evaluate the performance of the proposed V-FedMV, H-FedMV, and S-FedMV on a new multi-view sequential keystroke dataset collected from the BiAffect¹ study.

8.2 Experimental Setup

Data Preprocessing. In the experiment, we investigate a session-based mood prediction problem same as prior works in the BiAffect project [8], which utilizes features of three views in a session to predict a user’s mood score. However, our goal is to train a binary mood classification. To get the binary labels, after consulting with the professional experts, we label the sessions with the HDRS score between 0 and 7 (inclusive) as negative samples and those with the HDRS score higher than score 7 as positive samples.

However, the original input data is not naturally fitted our proposed V-FedMV and H-FedMV, because the original keystroke data is time-sequential format, but these methods are designed for feature matrices. To address the data format problem, we applied a GRU [10], which is a simplified version of Long Short-term Memory (LSTM) [24] to each view, respectively, to preprocess the data and extract feature embedding matrix as the input of V-FedMV and H-FedMV methods.

In this project, we use Keras with Tensorflow as the backend to implement the code. We use RMSProp [51] as the optimizer for GRU training. We retain sessions that any view contains the number of keypress between 10 and 100, and then we have 14,971 total samples. Besides, we set the batch size as 128, the epoch as 500, the learning rate from \( \lbrace 0.001,0.005\rbrace \), and the dropout from \( \lbrace 0.1,0.3\rbrace \). We use the validation dataset to select the optimal parameters and get the feature matrices which are the input of our V-FedMV and H-FedMV frameworks. Furthermore, the visualization of three views after data preprocessing is shown in Figure 5(a), demonstrating the original space of the input data of V-FedMV and H-FedMV.

Fig. 5.

View Figure

Hyperparameter Setting. In V-FedMV, we set up three clients, split the processed feature matrices according to the type of view, and make each client own one of three views. Besides, We set four clients in both H-FedMV and S-FedMV. In H-FedMV, we split the processed feature matrices, and make each client own the same number of samples with all three views. We also equally assign positive and negative samples to each client for H-FedMV. The data of S-FedMV is partitioned similarly to H-FedMV but the data is time-sequential.

For more details of hyperparameter setting, V-FedMV uses \( \beta _k \) as 4, \( \zeta _k \) and \( \eta \) from \( 2^0 \) to \( 2^5 \). In H-FedMV, we set the max communication rounds as 20, and the value of \( \beta _k^l \) as 4, and \( \zeta _k^l \), \( \eta ^l \) across different clients to be the same from \( 2^0 \) to \( 2^5 \). In S-FedMV, the max communication round is 150 for alphanumeric and special views, 170 for accel view. We set the local epoch from \( \lbrace 10,15\rbrace \), the dropout from \( \lbrace 0.1,0.3\rbrace \), the \( \beta _k^l \), \( \zeta _k^l \) and \( \eta ^l \) the same as H-FedMV. For all the experiments, the optimal parameters are selected by validation. In addition, we repeat all the experiments ten times with four metrics, i.e., accuracy, precision, recall, and F1 score. In addition, we also calculate the average and standard deviation.

8.6 Hyperparameter Analysis

We use the hyperparameter \( \zeta _1,\zeta _2,\zeta _3 \) to correspond with alphanumeric view, special view and accel view, respectively, which controls the balance between the three views. For instance, a large \( \zeta _k \) indicates a higher impact of the kth view of model training. In this experimental setting, we test each hyperparameter \( \in \lbrace 2^0,2^1,2^2,2^3,2^4,2^5\rbrace \). Note that while testing one hyperparameter, we fix the others as \( 2^3 \). We show the evaluation results with four metrics, i.e., accuracy, precision, recall, F1 scores shown in Figure 6. We find that the hyperparameters affect the accuracy and F1 score consistently, which shows a similar trend line for the same hyperparameter evaluation. In summary, while \( \zeta _1=2^4 \), \( \zeta _2=2^2 \), and \( \zeta _3=2^3 \), the model can achieve the best performance on accuracy and F1 score for both H-FedMV and V-FedMV approaches. While \( \eta =2^4 \), H-FedMV is better than V-FedMV, but V-FedMV is better than H-FedMV once \( \eta =2^5 \). Moreover, we also find the performance of the trained model is most sensitive to \( \zeta _1 \) and \( \zeta _2 \) and less sensitive to \( \eta \).

Fig. 6.

View Figure

9 RELATED WORK

In this section, we review the related work, which can be placed into three main categories: multi-view learning, federated learning, and federated multi-view learning.

Multi-view Learning: Sun et al. [49] provided a survey for multi-view machine learning and pointed out that multi-view learning is related to the machine learning problem with the data represented by multiple distinct feature sets. Xu et al. analyzed different multi-view algorithms and indicated that it was consensus and complementary principles that ensure their promising performance [56]. The aim of the consensus principle is to minimize the disagreement on multiple distinct views. A connection between the consensus of two hypotheses on two views, respectively, and their error rates was given by Dasgupta et al. [12]. The complementary principle means that multiple views can be complements for each other and can be exploited comprehensively to produce better learning performance for the reason that each view may contain some specific information that other views do not have. Wang and Zhou [52] demonstrated that the performance of co-training algorithms was largely affected by the complementary information in distinct views. These two principles are very important for multi-view learning and should be taken into consideration when designing multi-view learning algorithms.

Xu et al. also categorized the classical approaches of combining multiple views into co-training style algorithms, multiple kernel learning algorithms, and subspace learning-based approaches [56].

• Co-training style algorithm: Blum and Mitchell [4] proposed the original co-training algorithm to solve semi-supervised classification problems. First, two classifiers are trained separately on each view, and then each classifier labels the unlabeled data that are then added to the training set of another classifier. Besides, Kumar and Daumé [35] extended the idea of co-training to an unsupervised setting. Since co-training algorithms usually separately train the base learners, it can be seen as a late combination method.

• Multiple kernel learning algorithm: Combining different kernels is another way to integrate multiple views and can be regarded as an intermediate method for the reason that kernels are integrated just before or during the training phase. These methods include linear combination methods [29, 36] and nonlinear combination methods [11].

• Subspace learning-based approach: In the subspace learning-based approach, there is an assumption that multiple views are generated from a latent subspace. This approach can be regarded as a prior combination of multiple views, and the goal is to obtain the latent subspace. Canonical correlation analysis (CCA) [25] is a classical subspace learning-based approach that can be applied to the datasets containing two views. Besides, it can be extended to cope with datasets represented by more than two views [33] and to kernel CCA [20].

Federated Learning: Federated Learning (FL) was proposed by McMahan et al. [44]. It is a collaborative machine learning paradigm for training models based on locally stored data from multiple organizations in a privacy-preserving way. Yang et al. gave a comprehensive survey for FL [58], which introduced horizontal federated learning, vertical federated learning, and federated transfer learning.

• Horizontal federated learning: In horizontal federated learning, or sample-based federated learning, the data sets share the same feature space but differ in sample ID space [58]. Smith et al. proposed a novel framework for federated multi-task learning and considered high communication cost, stragglers, and fault tolerance in the federated environment for the first time [48]. Bonawitz et al. designed a secure aggregation scheme that allows a server to securely compute users’ data from mobile devices [5].

• Vertical federated learning: In vertical federated learning, or feature-based federated learning, the data sets share the same sample ID space but have different feature spaces [58]. Some algorithms and models have been proposed for vertical federated learning. [21, 31, 46] are about secure linear regression. Du et al. defined two types of secure two-party multivariate statistical analysis problems: linear regression and classification problem. Secure methods are also proposed to solve these two problems [15]. Liu et al. proposed asymmetrical vertical federated learning and showed the way to achieve the asymmetrical ID alignment [41].

• Federated transfer learning: In federated transfer learning, the data sets are different in both samples and feature space [58]. Liu et al. proposed an end-to-end approach to the FTL problem and demonstrated it was comparable to transfer learning without privacy protection [40]. A federated transfer learning framework for wearable healthcare was proposed in Reference [9].

Federated Multi-view Learning: Federated learning with multi-view data is currently less well explored in the literature. Flanagan et al. integrated multi-view matrix factorization with a federated learning framework for personalized recommendations and introduced a solution to the cold-start problem [19]. Huang et al. proposed FL-MV-DSSM, which is a generic content-based federated multi-view framework for recommendation scenarios and can address the cold-start problem [28]. Xu et al. [57] extended the DeepMood model [8] to a multi-view federated learning framework that suits the horizontal case. Feng et al. extended the idea of multi-view learning and proposed MMVFL that enables label sharing from its owner to other participants [18]. MMVFL suits for vertical setting and it can deal with multi-participant and multi-class problems while the other existing VFL approaches can only handle two participants and binary classification problems. Kim et al. proposed a federated tensor factorization framework for horizontally partitioned data [34]. To sum up, today’s federated multi-view learning is mainly applied to recommendation systems or tenor data, or only consider the horizontal or vertical situation, rather than consider the two together.

10 CONCLUSION

In this article, we proposed a generic multi-view learning framework by using federated learning paradigm for privacy-preserving and secure sharing of medical data among institutions, which can well protect patient privacy by keeping the data within their own confines and achieve multi-view data integration. Specifically, we investigated two types of multi-view learning (i.e., vertical and horizontal data integration) in the setting of federated learning based on different local data availability, and developed vertical federated multi-view learning (V-FedMV) and horizontal federated multi-view learning (H-FedMV) algorithms to solve this problem. Moreover, we adapted our model to deal with multi-view sequential data in a federated environment and introduced a federated multi-view sequential learning method (S-FedMV). Extensive experiments on real-world keyboard data demonstrated that our methods could make full use of multi-view data and obtain better classification results as compared to local training. Moreover, the result of S-FedMV is comparable with the result of the centralized method that cannot protect the privacy of sensitive data and thus showing the effectiveness of our federated method.

ACKNOWLEDGMENTS

We thank CAAI-Huawei MindSpore Open Fund and Huawei MindSpore platform for providing the computing infrastructure.