Predicting New-Onset Psychiatric Disorders Throughout the COVID-19 Pandemic: A Machine Learning Approach

Concerns have been raised regarding the mental health consequences of the long-lasting COVID-19 pandemic (1). Studies estimating self-reported symptoms of mental disorders, assessed by online surveys worldwide, have revealed a perceived decline in the general population’s mental health, at least in the short term. The pooled prevalence rates of clinically significant anxiety, depression, distress, and posttraumatic symptoms reached approximately 30%−40%, which were higher than rates typically observed in the general population. Some subgroups appeared to be particularly affected, such as health care workers (HCWs), patients with chronic general medical conditions or mental disorders, or people who had contracted COVID-19 (2–5).

Multiple potential predictors of unfavorable mental health consequences in the general population during the pandemic have been identified, including sociodemographic factors, such as being female (5–9) and being younger (6, 8, 10); individual features, such as having lower adaptive mechanisms to stress (11, 12); and pandemic-related stressful events (6).

In this scenario, some aspects have received less attention. It is unclear to what extent people without preexisting mental disorders have developed possible new-onset psychiatric disorders (PsyDs) during the pandemic. Limited studies on this issue are available, performed with various methodologies and providing conflicting results (13–16). Furthermore, most data were cross-sectional and collected only at an early stage of the COVID-19 outbreak. Finally, most studies examined sets of potential predictors employing traditional statistical analyses, which do not help researchers understand which factors have the most influence on mental health outcomes or their possible interactions. Only a few studies have used a machine learning (ML) approach (9, 11, 17), which is useful to detect patterns or predictive models in large and highly composite data samples (18, 19).

The study reported here was based on data from two waves of an online survey that we distributed among the Italian general population in two periods of the pandemic. Its purposes were to longitudinally evaluate the rates of new-onset psychiatric disorders (PsyDs) in Italian adults without preexisting mental disorders and to develop a predictive ML model of the new onset of at least one PsyD in subsequent independent samples of Italians during the pandemic. In our survey, we used screening tools based on DSM diagnostic criteria to maximize the specificity of the detection of PsyDs and to minimize the risk of classifying as “pathological” the common reactions of increased distress that can be part of a physiological emotional response to an unexpected global crisis. However, due to the self-reported nature of the screening tools, we considered the diagnoses of new-onset PsyDs as provisional (PPsyDs). To the best of our knowledge, no other published studies have addressed these aims.

Methods

Procedures

We distributed an online self-report survey among the Italian general population in two subsequent periods during the COVID-19 pandemic—namely, from May 18 to June 20, 2020 (first wave), and from September 15 to October 20, 2020 (second wave). These two waves were part of an ongoing longitudinal study aiming to monitor the Italian general population’s mental health up to 2 years from the beginning of the pandemic via subsequent online surveys launched approximately every 3 months. The procedure described below was used for the first two waves and will be used afterward in all waves of the study.

The survey was conducted using the SurveyMonkey platform, an online survey provider (http://www.surveymonkey.com) and was published and shared via social media (Facebook, Instagram, LinkedIn, and WhatsApp). We advertised the survey through the official Facebook and LinkedIn pages of Humanitas University and Humanitas San Pio X Hospital in Milan. We shared the survey link on our personal Facebook, Instagram, or LinkedIn pages and with a snowball spreading technique through personal contacts on WhatsApp. People who were ages 18 and older and who were not HCWs were invited to complete the survey on a voluntary and anonymous basis. Before starting each survey, all participants were required to provide written informed consent.

At the beginning of each survey, participants were asked to enter a few letters and numbers in response to standardized hints identical across the surveys (e.g., “Please enter the first two letters of your mother’s name”) to create a unique anonymous identifier for tracking respondents longitudinally during the study. Moreover, at the beginning of the second-wave survey, participants were asked if they had participated in the first. For each participant, the recorded data were saved and managed in accordance with the European regulations for privacy and protected health information. All the information is available on SurveyMonkey privacy notice section (www.surveymonkey.com/mp/legal/privacy). The entire longitudinal study was approved by the Ethics Committee of Humanitas Research Hospital.

Participants

A total of 3,532 participants provided their informed consent. For the aims of the study, we included in the analyses only participants who met the following criteria: completed the entire survey; declared never having had clinician-diagnosed lifetime mental disorders before the pandemic; declared never having contracted COVID-19 (because we aimed to exclude highly specific COVID-19–related risk factors), and, for participants in the second wave, not having participated in the first. Therefore, our final sample included 500 participants in the first wave and 236 participants in the second wave. The participant selection process is reported in Figure 1.

Measures

The survey consisted of two sections. One section included a series of ad hoc questions to collect participants’ sociodemographic characteristics and a wide range of individual information, such as lifestyle, personal relationships, general medical and psychiatric history, occupation, and usual disposition toward multiple aspects of daily life. Furthermore, a provisional diagnosis of obsessive-compulsive disorder (OCD) was assessed via ad hoc questions mirroring those of the Mini-International Neuropsychiatric Interview (MINI) for DSM-5 (20) and corresponding with the criteria outlined therein. The DSM-5 diagnostic rule was followed to identify a provisional diagnosis of OCD. The other section included a series of validated self-report screening tools (Italian-language version). We describe below only the tools that we used for the specific aims of this study (the entire list of questions and tools included in the survey is available on request):

Patient Health Questionnaire (PHQ).

The PHQ has modules for depression, anxiety disorders, and panic disorder (21). DSM-IV criteria–based diagnostic algorithms identify provisional diagnoses for selected DSM-IV disorders. Sensitivity and specificity (95% confidence interval [CI]) for major depressive disorder (MDD) were 0.73 (95% CI=0.59–0.87) and 0.98 (95% CI=0.9−1.00), respectively, in the original study (21), whereas recent estimates from multiple studies placed pooled sensitivity and specificity at 0.61 (95% CI=0.54–0.68) and 0.95 (95% CI=0.93–0.96), respectively (22). Sensitivity and specificity for any anxiety disorder (AD) were 0.63 (95% CI=0.53–0.63) and 0.97 (95% CI=0.95–0.99), respectively; for panic disorder, they were 0.81 (95% CI=0.69–0.93) and 0.99 (95% CI=0.98–1.00), respectively (21).

PTSD Checklist for DSM-5 (PCL-5).

The PCL-5 is a 20-item questionnaire based on DSM-5 posttraumatic stress disorder (PTSD) symptoms (23, 24). The DSM-5 criteria–based diagnostic algorithm identifies a provisional PTSD diagnosis with a sensitivity and specificity of 0.84 and 0.68, respectively (24, 25). In the instructions for completing the PCL-5, we clearly stated that any “very stressful experience” referred to should have been related to the COVID-19 pandemic.

Brief Resilience Scale (BRS).

The BRS includes six questions measuring resilience as a process, namely as “the ability to bounce back or recover from stress” (26, 27). Each response can vary from 1, strongly disagree, to 5, strongly agree, providing a sum-total score ranging from 6 to 30. Dividing the total score by the total number of questions answered, a final score is obtained, indicating low (1.00–2.99), normal (3.00–4.30), or high (4.31–5.00) resilience.

ML Methodology

ML is a fast-growing field at the crossroads of computer science, engineering, and statistics that gives computers the ability to learn without being explicitly programmed. ML techniques use training examples to create algorithms able to provide the best possible prediction when applied to new cases whose outcome is still unknown. In this study, for variables that were potentially useful as predictors in the ML model, we included all the individual information concerning aspects preceding the pandemic or representing its direct consequences, such as pandemic-related personal experiences—namely, the 46 variables that are reported in Tables 1–4.

Among the several available ML techniques, we applied gradient boosted decision trees (GBDT) in this analysis. Several decision trees are iteratively built, each consecutively trained to reduce the misclassification of the previous decision trees, and the final prediction is the result of a weighted sum of the prediction performed by all decision tree models, which can be as many as hundreds (28).

The development of an algorithm with this ML technique requires the definition of several hyper-parameters. Each configuration of these hyper-parameters leads to different predictive performances of the algorithm by differently tuning the training process. To identify such optimal hyperparameter configuration, at first 40 random hyper-parameter configurations were attempted, and then 60 further configurations were progressively estimated with a Bayesian optimization approach.

Because the aim is to identify the configuration that produces an algorithm with the best possible performance when applied to cases that are not used during training, stratified 10-fold cross-validation was applied in 100 hyperparameter configurations. In cross-validation, the training sample is divided into several folds of cases that are separated from the training process, with training iteratively performed with the remaining cases. After the training, the algorithm is finally applied to the separated cases. The hyper-parameter configuration that demonstrated the best average cross-validated area under the receiving operating curve (AUROC) was considered the best configuration and chosen to be used for the algorithm. The AUROC value is 0.5 when the algorithm makes random predictions, and 1 when it is always correct in making predictions.

An a priori selection of the predictive variables to be included in the algorithm is expected to improve the algorithm performance by including only relevant input variables and excluding irrelevant and redundant ones. We used the minimum redundancy, maximum relevance (mRMR) approach (29) in this study. This technique ranks all available predictive variables in order of importance by simultaneously taking into consideration the association with the output variable (maximum relevance) and the association with other predictive variables (minimum redundancy). The above-mentioned hyper-parameter optimization and cross-validation procedure was performed 46 times, each time considering as predictors a subset of size 1 to 46 variables (all variables) as indicated by the mRMR procedure and defining the best subset of the initial 46 predictive variables based on average cross-validated AUROC. (A detailed description of the ML methodology summarized above is presented in the online supplement.)

Training and testing protocol.

For training and cross-validation of the final algorithm, we used only observations from the first-wave survey. We used the observations from the second-wave survey as an independent test set to investigate the predictive performance of the finalized version of the algorithm. The algorithm initially outputs a continuous prediction (range, 0–1; the closer to 1 the higher the predicted probability of having at least one new-onset PPsyD to which a threshold is applied to obtain the final dichotomous prediction. Different threshold values can result in different predictive performances in terms of sensitivity and specificity. In this study, we chose the threshold value that minimized the achieved difference between sensitivity and specificity of the cross-validated predictions in the training data set. This value was applied to obtain the final prediction in the test data set.

Estimating the importance of predictive variables.

As with most ML techniques, the inherent complexity of the GBDT model did not allow us to interpret how the algorithm estimated the output starting from the included features. To overcome this limitation, there are techniques to make ML algorithms more interpretable. In this study, we applied the SHapley Additive exPlanations (SHAP) technique (30), which is based on the game theory approach. For each prediction, a SHAP value is assigned to each variable. The larger the absolute SHAP value of a certain variable, the larger the contribution of that variable in determining that prediction in a specific case. Moreover, in this study, a positive SHAP value indicated a contribution toward an increased risk of at least one new-onset PPsyD, whereas a negative SHAP value indicated a contribution toward a reduced risk.

The absolute average of the SHAP values observed for all cases in a data set can be used to identify the overall importance of each variable in the predictive algorithm. Plotting the values of a variable against the associated SHAP values can be used to visualize the relationship between the variable and the risk of at least one new-onset PPsyD that has been modeled by the algorithm. The SHAP approach has been applied distinctly in the observation collected in the first-wave survey (training data set) and the second-wave survey (test data set).

Results

Data on the geographical distribution of participants are summarized in Table S1 in the online supplement). Descriptive statistics of the variables included as potential predictors are presented in Tables 1–4. Eighty participants (16.0%) in the first wave and 44 (18.6%) in the second wave met the criteria of at least one new-onset PPsyD. The distributions of new-onset PPsyDs did not significantly differ between the two waves (Table 2).

Performance of ML Algorithm

Among the 46 variable subsets (from size 1 to all variables) indicated by the mRMR procedure, the subset for which the hyper-parameter optimization and cross-validation procedure in the training data set resulted in the best cross-validated AUROC was the subset that included 10 variables, with a AUROC of 0.8218. These variables encompassed the eight variables that remained in the final model (Figures 2 and 3A) and two variables concerning morbidity for general medical conditions and pandemic-related changes in drinking alcohol. The threshold value that minimized the difference between sensitivity and specificity in the cross-validated predictions was 0.191. Applying this threshold to the cross-validated predictions, a sensitivity of 73.33%, a specificity of 74.22%, a positive predictive value of 39.51%, and a negative predictive value of 93.12% were observed. This hyper-parameter configuration was subsequently used to train the final model by using the entire training set without cross-validation.

When the final model was tested using data from the second wave (test data set), the AUROC was 0.7847 (95% bootstrap CI=0.7071–0.8575). Considering the categorical predictions generated with the threshold identified above, our results indicated an average sensitivity of 70.00% (95% bootstrap CI=55.00%–82.50%), an average specificity of 72.99% (95% bootstrap CI=66.09%–79.31%), an average positive predictive value of 37.33% (95% bootstrap CI=30.21%–45.16%), and an average negative predictive value of 91.37% (95% bootstrap CI=87.5%–95.1%).

Importance of Predictive Variables

The final predictive model included only eight of the 10 initial variables, because two variables were discarded during the training process (i.e., morbidity for medical conditions and pandemic-related changes in drinking alcohol).

The importance of the eight variables included in the final model was analyzed with the SHAP technique in data from both the first wave (training data set) and the second wave (test data set). Figure 3A presents the average of the absolute SHAP values for each variable obtained in the second wave, ordered by their relevance to the model. Specifically, it visualizes the average contribution of each variable to the risk estimation of having at least one new-onset PPsyD in the sample. The largest contributions were provided by item 3 of the BRS, employment status, and two variables concerning levels of perceived stress due to pandemic-related conditions. The variables displayed similar average contributions to the model in both waves, except for a partial increase in the contribution of employment status in the second wave, determined by a higher prevalence of “being an undergraduate student” in the second wave, compared with the first wave. “Having experienced a loved one’s hospitalization” made the smallest average contribution, thereby playing only a marginal role in the model.

The SHAP values obtained in the first wave are presented in Figure 2. Levels of item 3 of the BRS plotted against the associated SHAP values in the second wave are presented in Figure 3B, as an exemplifying visual representation of the relationship between a variable and the risk of having at least one PPsyD that has been modeled by the algorithm. (All the other plots in the second wave are presented in Figure S1 in the online supplement.) The plots in the first wave are available on request.

Discussion

We have presented data from two waves of an online survey that we distributed among the Italian general population in two periods during the COVID-19 pandemic: in the middle of May 2020 (first wave, immediately after the Italian lockdown) and in the middle of September 2020 (second wave, after a few months with looser restrictions on activities and personal movement).

The novelty of this study was the estimate of new-onset PPsyDs over the first 8 months of the pandemic in Italians without preexisting self-reported PsyDs and the development of an ML model able to predict the new-onset of least one PPsyD in an independent sample of participants in the second wave after it was trained on data from the first wave. The ML approach allowed us to consider a large set of personal and psychosocial variables in a complex high-stress context and identify variables that represented predictors of the emergence of new-onset PPsyDs in the general population. No other published studies, to our knowledge, have addressed these aims.

New-Onset PPsyDs

We found that 16.0% of participants in the first wave and a further 18.6% in the second met the provisional diagnosis of at least one new-onset PsyD (PPsyD), as assessed via self-report screening tools based on DSM diagnostic criteria. Concerning specific disorders, the majority of PPsyDs were of MDD and any AD, followed by OCD, PTSD, and panic disorder.

Unfortunately, the lack of epidemiological data concerning the incidence of common mental disorders in Italy before the pandemic did not allow us to provide incidence comparisons with our results. However, our findings suggested a possible overall increase in PsyDs among the Italian general population, compared with the 12-month or lifetime prevalence of common mental disorders estimated in Italy before the pandemic (31–33).

Data from other countries are limited and heterogeneous in methods and results. In two Canadian studies, the rates of clinically significant self-reported anxiety, depression (13), and OCD (14) symptoms at the beginning of the pandemic were approximately 35%, 49%, and 60%, respectively, in people without prior self-reported psychiatric diagnoses. Similar to our study, these results pointed to a substantial burden of new psychiatric symptoms during the pandemic in people who were previously mentally healthy. In both studies, however, the psychiatric assessment methodology was highly inclusive, making it difficult to estimate what portions of participants might have developed a “true” PsyD. Conversely, a Chinese study did not find any indication of self-reported anxiety, depression, or stress in a small sample of healthy controls (HCs), whereas it found a substantial rate (14%) of possible PTSD (15). However, the lack of information concerning the method of recruitment and assessment of HCs did not allow us to draw conclusions about these results. Finally, people without preexisting PsyDs who were participating in three Dutch cohort studies preceding the pandemic were found to have a pandemic-related increase in clinically significant self-reported psychiatric symptoms. However, the psychometric tools used did not provide cutoffs to identify PPsyDs (16).

Our results should be interpreted with caution for several reasons. Because we estimated PPsyDs via self-reported assessment, our findings are not directly comparable with the lower rates obtained via clinical interviews in the most comprehensive prepandemic epidemiological Italian study (31). The other two available prepandemic estimates (32, 33) were methodologically more similar to our study, because they provided PHQ-based rates of provisional MDD diagnoses, which were 2.5% (32) and 6% (33). These prepandemic rates were lower than those we found during the pandemic, even though the results are not fully comparable because of the use of different PHQ versions.

Concerning OCD screening, we created an ad hoc questionnaire for this study. Although it mirrored the questions of the MINI for DSM-5, its sensitivity and specificity were not previously tested.

Even though we sought to maximize the specificity of the psychiatric diagnoses by using DSM criteria–based assessments (21, 22, 25), we cannot rule out the possibility that at least some of the identified provisional diagnoses did not represent stable PsyDs but were only temporary states of intense distress or discomfort. Longitudinally tracking participants who met new-onset PPsyD criteria during the first or second wave in subsequent phases of our study will allow us to monitor the diagnoses’ stability. Finally, we cannot rule out the possibility that people who declared themselves to never have had clinician-diagnosed PsyDs could have had undiagnosed psychiatric conditions. Bearing in mind these limitations, the burden of new-onset PPsyDs among Italians appears to be worthy of attention.

ML Predictive Model

Our final best ML predictive model displayed a sensitivity of 70% and a specificity of 73% when tested in the second wave, suggesting a moderate performance in predicting new-onset PPsyDs in independent samples of Italians during the pandemic. These findings suggest that the same factors that had influenced the occurrence of new-onset PPsyDs at the beginning of the pandemic could have continued to maintain their influence in the subsequent months.

The final ML model encompassed eight variables. Among these, the largest contributions to the prediction were made by factors preceding the pandemic—dispositional traits of low resilience and being an undergraduate student—and another two factors directly related to the pandemic—namely, highly perceived stress in response to the possibility of spreading the infection to others and measures restricting personal movement.

Our findings are in line with the well-known influence of stress resilience on mental well-being and vulnerability to PsyDs (34), and they expand previous associations between lower psychological resilience, or coping capability, and higher emotional distress, anxiety, and depressive symptoms among Italians at the beginning of the pandemic (9, 12, 35).

We identified undergraduate students as the category most vulnerable to PPsyDs, whereas no other occupational status appeared to have relevance to the model. The predictive contribution of being a student increased in the second wave. This finding could be explained by the increased rate of students who participated in the second wave, compared with the first, and it provided additional support for the importance of this variable in the prediction. Our results expand previous preliminary associations between being a student and the significant perceived stress and depressive symptoms found among Italians in a very early phase of the pandemic (6). Multiple stressors with the potential of exerting detrimental effects specifically on students’ mental well-being have occurred during the pandemic. Distance learning, daily physical isolation, delays in academic activities, concerns over slowing down of their studies, suspension or cancellation of learning experiences outside their own city or abroad, and suspension of traveling to their city of origin if they were studying away from home are just a few examples. Therefore, it is conceivable that a burden of pandemic-related factors peculiarly involving students could have contributed to the emergence of new-onset PPsyDs in this specific portion of the younger population, which per se is typically at higher risk of developing many PsyDs, compared with older people (36). Our results highlight that the mental health of this population category should be monitored during public health emergencies, and large-scale supportive interventions should be offered at universities.

The other two identified variables suggest a predictive contribution of the subjective perceived stress in response to pandemic-related “inevitable” issues commonly shared across the population. Although improving one’s own emotional response to stressors clearly requires psychological interventions, implementing large-scale informative and supportive campaigns about these sensitive topics during public health emergencies could help people who are more susceptible to stress better overcome and sustain such challenges.

Finally, the ML model included other variables that contributed to the prediction to a much lesser extent. However, the role of living alone deserves mention, given this condition has previously been demonstrated to confer risk of developing common mental disorders, especially when associated with perceived loneliness (37). Our results consistently highlight that people living alone can be particularly vulnerable and deserve high-level attention during public health emergencies, when the deprivation of interpersonal communication and perception of social isolation are likely to increase. Similarly, the predictive contribution of ceasing physical activity expands previous associations between reduced physical activity and reduced psychological health found among Italians at the beginning of the pandemic (38) and is in line with the well-known positive effects of regular exercise on mental well-being and stress response (38). Our results emphasize that public campaigns to promote maintenance of regular physical activity, even during public health emergencies, could be important to the population’s mental well-being.

The main strengths of this study were in providing a follow-up of the Italian population’s mental health and in developing a predictive ML model of PPsyDs applicable to independent samples of Italians during the pandemic. Furthermore, we used techniques helpful for selecting variables with the maximum relevance and the minimum redundancy to the prediction. Overall, the ML approach allowed us to identify the most important predictors from a large set of potential predictive variables, offering more reliable and significant results than traditional statistical analyses. Traditional analyses present limitations when used for analyzing this type of data set, possibly leading to inaccurate conclusions. The issue of multiple comparisons, which inflate type I errors, or the inability to take into account multiple possible interactions among variables are just a few examples of those limitations. This strategy was of special interest in the pandemic scenario because it could contribute to selecting priority targets of attention or intervention.

Nonetheless, there are limitations in addition to those already mentioned. The sample was small. Many participants were excluded because they declared that they had experienced previous PsyDs or because they did not complete the entire survey, and the number of participants declined in the second wave. The high participation of people with PsyDs was likely due to the recruitment methodology, which was mainly based on official institutional sites. This approach could explain why most participants were from northwestern Italy, which prevented us from including geographic origin in the analyses and rendering our results not generalizable to all Italians. Given the missing data, we could not include in the analyses variables concerning the relationship with a partner and the fear of contracting COVID-19 or dying from COVID-19 infection. Moreover, because the survey was limited to adults, we did not have information concerning younger students. We included only the BRS to assess resilience. It is well known that resilience is a multifaceted construct, and its complexity cannot be captured by a single questionnaire (39, 40). The nature of our study, which was based on online surveys and aimed to collect a high number of potential predictors of mental health outcomes, required simplifying the assessment. Therefore, we used a single selected scale at the expense of multiple comprehensive questionnaires to assess a single construct, such as resilience. Although the measurement of “the ability to bounce back or recover from stress” (27) appeared to be suitable to the pandemic context and the aim of the study, this simplification excluded a more comprehensive perspective of resilience as a dynamic process of adaptation to stressors, involving biological, psychological, and social components (39).

Finally, we focused on a subset of PsyDs with a high prevalence and thereby detectable even with a moderate sample size. We did not include other disorders, such as eating disorders, somatic symptom and related disorders, or substance use disorders. To fill this important gap, and especially to detect disorders that can have high incidence and impact in the general population, such as substance use disorders, future follow-up surveys should investigate the potential new emergence of a broader range of diagnoses. Therefore, future studies are warranted to confirm and expand our results.

Conclusions

We found substantial rates of new-onset PPsyDs over the first 8 months of the pandemic in Italian adults who declared themselves to never have previously had PsyDs. We identified a predictive ML model of the occurrence of at least one new-onset PPsyD in independent samples during the pandemic. Our results could highlight modifiable vulnerability factors that are suitable to be targeted by public campaigns or interventions to mitigate the pandemic’s detrimental effects on mental health.