Visual Scanning of Facial Expressions in Schizophrenia

Individuals with schizophrenia are impaired in aspects of interpersonal communication, one of which is the ability to identify emotional expressions in faces.¹ Studies in face perception have indicated the importance of perceiving both individual facial features and the configuration formed by these features.² The configuration of a face refers to relations among its internal features, its global structure that forms a gestalt. This study addressed the possibility that schizophrenic patients' deficits in identifying facial emotions arise in early stages of visual processing and are related to difficulties in perceiving facial configurations.

Many investigators have attempted to understand visual perceptual processes by recording eye movements.^3,4 For instance, it has been suggested that eye movements provide an overt measure of how people direct and shift their attention while reading text and perceiving objects. Eye movements fall into two main categories: fixations, which are periods of relative eye stability, and saccades, which are rapid jumps of the eye from one point of the visual field to another. Saccades change the direction of our eye so that the fovea (or center of vision) is directed at the object of perception.

Examinations of eye movements, and specifically visual scanning in free-viewing tasks, have shown several different types of abnormalities in patients with schizophrenia. These patients show restricted visual search when looking at geometric shapes⁵ and show extensive staring, extensive scanning, and a focus on irrelevant details when viewing complex scenes and faces.^6–9 These deficits reflect problems in the scope and focus of visual search (scanning) while viewing complex stimuli.

In a previous study,¹⁰ we observed that schizophrenic patients made fewer short-duration fixations (≤50.1 ms) while viewing faces compared with a clinical control group with cocaine use disorder. One hypothesis that we proposed in that report was that eye movements of less than 50.1 ms might be involved in perceiving the face as a whole (facial configuration), whereas longer eye movements might be required to perceive individual facial features. This hypothesis was based on findings that subjects can perceive certain patterns or elements in a pattern within 40 to 50 ms by means of an automatic or parallel process, referred to as preattentive search.¹¹ Preattentive search is distributed over the whole image; that is, patterns can be discriminated very rapidly over large areas.^11,12 By contrast, attentive search requires longer scrutiny (>50 ms) of individual elements in the pattern and is carried out by a serial search process.¹¹ Because investigators of face perception draw a distinction between perception of the face as a whole (configural processing) and perception of individual facial features (part-based or component processing), we hypothesized that preattentive search might be involved in configural processing, whereas attentive search was involved in processing parts of the face.

Researchers have long suggested that patients with schizophrenia perceive objects and scenes in a fragmented manner and that they are impaired in forming perceptual gestalts.^13,14 It is possible, therefore, that our findings of fewer short-duration eye movements (≤50.1 ms) in the patients might indicate problems in configural processing of faces. The objective of this study was to test whether abnormal visual scanning of faces in patients with schizophrenia is related to their problems in perceiving configurational information.

To address this question, we presented faces in an upright and inverted orientation. There is a well-established finding that inverting a face disrupts its recognition.¹⁵ This is thought to occur because inversion distorts the spatial arrangement of internal facial features. We reasoned that if eye movements of 50.1 ms or less are involved in scanning configurational information, then distorting facial configurations by inversion would affect these eye movements in normal subjects. In contrast, if schizophrenic patients' scanning of faces is not dependent on configural processing, then their eye movements that are 50.1 ms or less should be unaffected by inverting faces.

METHODS

Subjects

Sixteen patients who fulfilled DSM-IV criteria for schizophrenia and 10 comparison subjects participated in this study. A diagnosis of schizophrenia was made after an interview with the patient and a review of his or her chart. During this interview, the severity of the patient's symptoms was determined in consensus by two raters (R.B.R. and B.L.S.) using the Brief Psychiatric Rating Scale (BPRS)¹⁶ and the Scale for the Assessment of Negative Symptoms (SANS).¹⁷ Interrater reliability for these two raters was 0.97 for the BPRS and 0.76 for the SANS. The score for each item in these scales was obtained through consensus by these two raters.

The patients' mean age was 40.8±5.4 years (values reported as mean±SD) and their educational level was 12.6±1.4 years. All patients were treated with antipsychotic medication at the time of testing. Their mean daily dose of medication in chlorpromazine equivalents was 579.70±575.5 mg. Their average length of illness was 15.6±5.61 years. Patients' mean score was 54.38±11.35 for the BPRS and 49.8±14.6 for the SANS. Patients were free of major medical and neurological disorders. Table 1 presents individual characteristics for each patient.

Comparison subjects were paid volunteers who were recruited from the hospital staff and community. They were screened to exclude subjects with medical or psychiatric problems. The mean age of comparison subjects (35.2±9.94 years) did not differ reliably from the patients' mean age (t=1.85, df=24, not significant). The mean educational level of these subjects (14.5±2.9 years) was significantly higher than that of the patients (t=–2.33, df=24, P<0.029). All subjects had normal or corrected-to-normal vision.

Procedures

All subjects signed an informed consent form before their participation in the study. Subjects viewed six slides of faces with an emotional expression from the collection of Ekman and Friesen.¹⁸ Three faces were presented in an upright orientation and three in an inverted orientation, with each slide shown for 20 seconds. After 20 seconds, subjects were asked to identify the facial expression (happy, sad, or fear).

Eye movements were recorded while subjects viewed the slides, by use of an ISCAN pupil/corneal infrared reflection system (Burlington, MA). Each subject rested his or her chin in a chin rest and leaned the forehead against a padded bar to stabilize the head. Slides were presented on a television screen at a distance of 28 inches from the chin rest. The viewing screen was 12.5×17 inches (35.95° horizontally, 28.58° vertically). An infrared light source was directed at the right eye, producing a reflection of the pupil and cornea that was recorded by a camera to measure the eye's position. A digitized version of the vertical and horizontal eye position of the right eye was recorded every 16.7 ms (60 Hz). Visual scanning scores were derived from the first 10 seconds of the 20-second recording period with blinks removed.

The scanning data were analyzed according to the following algorithm: 1) the amount of displacement per unit time (16.7 ms) was computed by applying the Pythagorean theorem to the x and y coordinates of the eye position, using the diagonal of the eye movement as the total displacement between two consecutive points; 2) if the displacement was 0.75° of visual arc or less, the eye movement was classified as a fixation; if it exceeded 0.75° of visual arc, the eye movement was classified as part of a saccade (error of resolution 0.10°–0.50° visual arc); 3) the numbers of fixations and saccades that occurred on consecutive points were counted to yield the duration of the eye movement; and 4) the numbers of fixations and saccades that were 50.1 ms or less were counted and the numbers of fixations and saccades that were longer than 50.1 ms were counted, yielding four eye movement measures. For each measure, the mean number of eye movements was calculated across the three upright faces and the three inverted faces.

RESULTS

Table 2 presents the mean number of saccades and fixations for the patients and comparison subjects at each level of duration. Saccades and fixations were analyzed separately. The analysis for each was a 2 (group: patient and comparison) by 2 (face orientation: upright and inverted) by 2 (duration: 50.1 ms or less and greater than 50.1 ms) mixed analysis of variance (ANOVA). The first variable is a between-subjects variable and the latter two are within-subject variables.

The ANOVA on saccades yielded a main effect for face orientation (F=77.69, df=1,23, P<0.001) and a significant three-way interaction of group by face orientation by duration (F=8.97, df=1,23, P<0.01). The interaction revealed that the effects of facial orientation on saccades of ≤50.1 ms were different from those of >50.1 ms. Specifically, a 2 (group) by 2 (face orientation) ANOVA on saccades ≤50.1 ms yielded a significant interaction of group by face orientation (F=5.47, df=1,24, P<0.03). These results showed that normal subjects generated fewer short-duration saccades when the face was inverted compared with when it was upright. In contrast, the number of saccades ≤50.1 ms generated by the patients did not differ for upright and inverted faces. The 2 (group) by 2 (face orientation) ANOVA on saccades >50.1 ms yielded no significant effects. Analyses performed on fixations yielded no significant main effects or interactions.

One of the objectives of this work was to determine whether patients' difficulty in identifying facial expressions was related to their visual scanning abnormalities. As in prior studies, we also observed that patients with schizophrenia were less accurate in choosing the correct emotion shown on the face (67%) relative to comparison subjects (87%; t=–2.38, df=24, P<0.03). However, Pearson product-moment correlational analyses showed that there was no significant relationship between the patients' ability to identify facial emotion and the number of fixations or saccades (≤50.1 ms or >50.1 ms, all P >0.15).

In addition, we examined the relationship between eye movement measures and the severity of the patients' symptoms by using the BPRS and SANS. These analyses yielded no significant correlations between total scores on these scales and either saccades or fixations at each level of duration.

Finally, because all patients in the study were receiving medications, it is important to point out that the patients' daily dose of antipsychotic medication in chlorpromazine equivalents did not correlate with any eye movement measure or with their ability to judge facial emotion.

DISCUSSION

This study was designed to test whether abnormal scanning of faces in patients with schizophrenia was related to scanning of facial configurations. We presented faces in upright and inverted orientations based on the assumption that inverting faces distorts the integrity of internal features, thereby distorting the configuration formed by these features. The main finding was that normal subjects generated more saccades of 50.1 ms or less to upright than to inverted faces, whereas for the patients, the number of saccades did not differ by facial orientation. Moreover, the different pattern for patients and comparison subjects was observed only for saccades that were 50.1 ms or less, not for those that were longer than 50.1 ms.

We used eye movement recordings in this study to examine whether the patients' difficulty in identifying facial emotion was due to their abnormal scanning of faces. However, we and others⁷ did not find a correlation between judgments of facial emotion and eye movement measures. Two other findings with schizophrenic patients suggest that deficits in visual scanning are unrelated to deficits in judging facial emotions. First, our previous report¹⁰ showed that patients with schizophrenia and those with cocaine abuse differed in visual scanning of faces even when their emotion recognition performance was matched. The second finding is that inverting a face has different effects on eye movements than it does on verbal judgments about facial emotion. We have observed over several facial processing tasks (e.g., emotion recognition, identity recognition, and face recognition memory) that distorting facial gestalts influences the patients' verbal judgments in the same way it does control subjects' judgments (Schwartz et al., manuscript in preparation). Together, these results suggest that visual scanning deficits in the schizophrenic patients are likely due to other abnormalities in cognitive processing.

We suggested that eye movements that are 50.1 ms or less are involved in a visual search of configurational information in faces, particularly preattentional search as defined above. Inverting faces from their upright orientation might lead to a breakdown in this rapid scanning process because subjects cannot depend on the familiar schema of the face. The finding that schizophrenic patients' visual scanning is unaffected by inverting the face suggests an impairment in search processes that are involved in configural processing. These hypotheses are based on the notion that preattentive search is involved in perceiving the image as a whole—that is, attention would be distributed over the face as a whole rather than directed to its individual features.

Although deficits in attentional search would be consistent with findings of attentional problems in these patients,¹⁹ this account of their visual scanning behavior is speculative. We currently are testing the relationship between some of the eye movement parameters studied here and measures of attention. In addition, the present study is limited by the small number of subjects. These results will need to be confirmed in a larger group. It is possible that with a larger sample of patients we would have shown a significant correlation between schizophrenic symptomatology and short-duration eye movements, which has been reported previously.¹

Eye movements can provide an overt constituent of attention and cognitive processing. The different pattern of eye movements for the patients and comparison subjects observed in this study might indicate differences in preattentional search strategies that are involved in perceiving facial configurations. These results suggest further that schizophrenic patients' deficits in scanning facial emotions arise in early stages of visual processing.

TABLE 1. Characteristics of the patient group

Enlarge table

TABLE 2. Mean number of saccades and fixations generated to upright and inverted faces

Enlarge table