Keywords

1 Introduction

We developed the pupil-corneal reflection-based, remote, head-free gaze detection systems [1, 2], which are based on the robust pupil detection method using the two light sources and the image difference method. In these systems, an optical system for detecting the pupils consists of a video camera and a double concentric circle near-infrared LED rings (inner and outer rings) light source. The inner and outer LED rings of the light source generate the bright and dark pupil images, respectively. The pupils are detected from the difference image created by subtracting the dark pupil image from the bright pupil image. The light source also elicits the corneal reflection image. The gaze points on a PC screen are determined by the relative position between the pupil and corneal reflection (feature points). However, when the users move their head, the pupil position varies between the bright and dark pupil images because of the acquisition time difference for both pupil images. Therefore, the image difference is performed after shifting the small areas including each pupil in the dark pupil image so that the corneal reflection in this dark pupil image becomes coincident with that in the bright pupil image [3]. The pupils generally become small when the illumination on the face is strong, e.g. by the sunlight. It is difficult to detect the small pupils in robust due to the similarity in image characteristics between the pupil and the glass reflections of the light source. In addition, when the users move their head quickly, this image difference method does not tend to function accurately, resulting in undetection or misdetection of the pupils. Furthermore, the corneal reflection is directly detected in the bright and dark pupil images but not in the difference image, the disturbance light source, e.g. the sun and the light source irradiating the near-infrared light. The source produces the false corneal reflection and glass reflection. To solve these problems, in the present study, we try to use a high-speed camera because the acquisition time difference is extremely short. In addition, a non-lighting image is obtained consecutively with the bright and dark pupil images, and then the corneal reflection is detected using the difference images obtained by subtracting the non-lighting image from the bright and dark pupil images. Finally, we propose a novel light source to remove the glass reflections and to increase the brightness difference between the bright and dark pupil. The experimental result show the effectiveness of the proposed methods and light source.

2 Optical System and Image Acquisition and Processing Algorithms for Detecting Pupil and Corneal Reflection

Figure 1(a) shows the optical system which consists of the high-speed camera (2,000 fps, 512 × 512 pix), the 16 mm lens, the LED light source, and the near-infrared pass filter (IR80). The camera images were able to be transmitted to the PC memory and were image-processed in the real time. The light source consisting of the near-infrared LEDs, which are arranged in a double concentric circle rings form, is attached to the camera. The inner rings (850 nm) and outer rings (940 nm) produce the bright and dark pupils, relatively, because the LEDs near the aperture of the camera brighten the pupil more than the LEDs far from the aperture and because the transmissivity of the eyeball medium for 850 nm is greater than 940 nm. The previous light source had used 5ϕ shell type of LEDs. To increase the brightness of the bright pupil, we changed from 5ϕ to 3ϕ (Fig. 1(a)) because the center of the LEDs approaches the aperture more. However, depending on glasses, the glass reflection of the outer ring showed a doughnut shape because the radius of the ring was long, whereas the glass reflection of the inner ring shows a smaller filled circle. Their shape difference caused the failure of the cancellation of the glass reflection images and retained the wrecks of the glass reflections in the difference image. To remove the wrecks, we developed the small light source having the smaller radius of outer ring (each part of the light source in Fig. 1(b)). The cancellation of the glass reflections was successful. However, since the number of LEDs decreased, the irradiation power to acquire the dark pupil image decreased. This caused the difficulty of the pupil detection. To solve this problem, we developed the new light source that three of the above-mentioned small light sources were arranged horizontally, as shown in Fig. 1(b). The inner ring of the center small light source, which was attached to the camera, flashed in the bright pupil acquisition frame to obtain the bright pupil image. The outer rings of the three small light sources are flashed simultaneously and evenly in power in the dark pupil image acquisition frame. Here, the outer ring of the center source functioned to cancel out the glass reflection produced by the inner ring of the same source. However, the dark pupil effect due to the outer ring was weak. So, the outer rings of the right and left small light source was used to enhance the dark pupil effect.

Fig. 1.
figure 1figure 1

(a) Optical system using high-speed camera (2,000 fps). The light source uses 3Ď• shell type of near-infrared LEDs. (b) Newly developed light source for glasses.

Full size image

The bright pupil image, the non-lighting image, and the dark pupil images were consecutively obtained in this order while turning the respective rings on and off as shown in Fig. 2. This figure also shows the new image processing algorithm for detecting the feature points. By subtracting (a) the non-lighting image from (b) the bright pupil image and (c) the dark pupil image, (d) the difference bright pupil image and (e) the difference dark pupil image were made, respectively. These subtractions theoretically remove the affections of the ambient lights and gives the bright and dark pupil images illuminated by only the light sources of the system ideally. Next, (f) the multiplicated image and (g) the divided image are produced. From these images, (i) the corneal reflection image and (j) the pupil image are obtained. The above-mentioned image processing algorithm was used in all experiments.

Fig. 2.
figure 2figure 2

Image processing algorithm using high-speed camera for detecting pupils and corneal reflections.

Full size image

3 Experiments

3.1 Effectiveness of the Use of High-speed Camera for Fast Head Movement

Three university students served as subjects. They did not wear glasses and were asked to move their head laterally at a speed of approximately 20 cm/s. Their face images were captured at 60 fps and 2,000 fps. The experiments were conducted under the fluorescent light in the room (approximately 2,000 lux). An incandescent lamp (200 W) was installed at the position of 1 m from the subjects to generate the false corneal reflection and to constrict the pupils. The purpose of the experiment was the verification for the effectiveness of the use of the high-speed camera. The light source using 5ϕ shell type of LEDs, which was similar to the light source shown in Fig. 1(a), was used. We examined the detection of the pupil and corneal reflection in the pupil image (Fig. 2(i)) and the corneal reflection image (Fig. 2(j)), respectively, by visual observation. In the present study, when the maximum connected pixel area was the true pupil, we judged that the pupil detection was successful (correct detection). If the detected connected area in the neighborhood of the pupil center was the true corneal reflection, we judged that the detection of the corneal reflection was successful.

Figure 3 shows the samples of the pupil image (Fig. 2(j)) at 60 fps and 2,000 fps. The many connected pixel areas appeared in the pupil image at 60 fps. In the corresponding image at 2,000 fps, only the pupils were embossed. The detection ratios of the pupil and corneal reflection showed both 0 % for all subjects at 60 fps, whereas the detection ratios of both feature points showed 98 % for subject A and B and 58 % for subject C at 2,000 fps. Thus the detection ratios were dramatically improved by the use of the high-speed camera. The detection ratios of the pupil and corneal reflection showed the same values for all subjects. This means that the true corneal reflection tended to be detected regardless of the existence of the false corneal reflection due to the incandescent lamp. This indicates that the false corneal refection were canceled out by the differentiation of the non-lighting image.

Fig. 3.
figure 3figure 3

Samples of the pupil image at 60 fps and 2,000 fps

Full size image

3.2 Real Time Pupil Detection Under the Direct Sunlight

A subject who did not wear glasses moved the head laterally under the room light condition (the fluorescent light and the sunlight from the shaded windows, approximately 700 lux) and under the direct sunlight condition (approximately 40,000 lux). The distance between the pupil and the camera was approximately 80 cm. The pupil detection was performed 192 times. Figure 4(a) and (b) show the time courses of the coordinates of the right and left pupil. The ratios that both right and left pupils were detected were 99.5 % and 68.8 % under the fluorescent light and direct sunlight conditions, respectively.

Fig. 4.
figure 4figure 4

Results of real time pupil detection

Full size image

3.3 Effectiveness of Newly Developed Light Sources for Glasses

We compared the detection ratios of the feature points between the previous light source (Fig. 1(a)) and the newly developed light sources for glasses (Fig. 1(b)) under the direct sunlight condition (approximately 40,000–45,000 lux). Four subjects who wore glasses were asked to fixate the center of the PC screen while moving their head laterally. Figure 5(a) and (b) show the detection ratios of the pupil and corneal reflection of the right eye under the direct sunlight condition, respectively. The averaged correct detection ratios of the left and right pupils were 68.1 % and 85.0 % by the previous and newly developed light sources, respectively. Those of the left and right corneal reflections were 66.1 % and 80.1 % by the previous and newly developed light sources, respectively. Thus the detection robustness for the feature points were improved by using the newly developed light source. However, the misdetection of the corneal reflection for subject C increased compared to the previous light source. This is because the glass reflection image existed in the neighborhood of the pupil center for subject C and the image were not able to be canceled out by differentiation of the non-lighting image.

Fig. 5.
figure 5figure 5

Detection ratios for pupil and corneal reflection of right eye by using the previous and newly developed light source under direct sunlight condition (40,000–45,000 lux).

Full size image

4 Conclusions

The high-speed camera was introduced to detect the pupil in robust even when the pupil is extremely small, the user moved the head quickly, and the user wears glasses. The new image processing algorithm for detecting the feature points under the face intense illumination was proposed. In addition, the light source was improved to remove the glass reflection and to increase the brightness difference between the bright and dark pupils. The experimental results showed that the detection robustness for the feature points was improved.