Keywords

1 Introduction

Eye tracking has been on the rise in user experience research, particularly in usability testing [1, 2]. Eye-tracking data can provide additional insight into the user experience. For instance, fixation duration captures how long a participant looks at something, while a gaze plot shows where the participant looked on a page over a period of time in a sequential pattern [3]. Eye tracking can help us understand what part of a user interface draws participants’ attention. However, the use of eye tracking in usability research is still a relatively new and growing field. It is not always easy to interpret eye-tracking data. For example, a long eye-fixation duration could reflect either confusion about or engagement in the material [3–5]. In a typical usability study, a moderator conducts a debriefing interview after the participant completes tasks. During the debriefing, the researcher probes the participant with questions that arise from the observation of the participant’s task performance. Analyzing eye-tracking data in conjunction with debriefing data in the context of the study may facilitate interpretation of the eye-tracking data and can lead to a more comprehensive picture of the participant’s behavior.

With the prevalence of the Internet, the U.S. Census Bureau is moving more and more towards self–administered surveys, from paper to a Web-based online mode. To conduct a Web survey, the Census Bureau typically informs a respondent via a letter that he or she has been selected to participate in the survey. In addition to information about the purpose of the survey, the letter contains information such as whether the survey is mandatory or voluntary, the estimated amount of time it will take to complete the survey, when responses to the survey are due, the URL of the online survey, along with user name and login information. Some of this information is also shown on the login page of the Web survey or can be found by clicking on keyword links located at the top or the bottom of the survey screen. In other words, some of the same information is presented in both the letter and the login page. The Census Bureau is interested in knowing which pieces of information respondents attend to and how they behave with the different mediums. This knowledge can have an important impact on the design of a Web survey and letter. This paper presents the results of a usability study on participant behavior of reading two different stimuli: a letter and a login page of a Web survey (hereafter referred to as Web site). We examined participants’ reading attention using eye-tracking technology and a follow up post-task debriefing question. We particularly investigated what parts of the letter or the Web site the participants saw, and what they would remember about what they had looked at and/or read.

2 Methodology

2.1 Study Design

We used a between-subjects design to investigate participants’ behavior of reading a letter or a Web site. One group of participants was exposed to only the letter, while the other group was exposed only to the Web site. The same post-task performance assessment was conducted with both groups. In a typical survey production setting, the respondent will first receive a letter inviting them to participate in a Web survey. However, for this study we did not want the participants to have been exposed to any information about the survey prior to the test, as we could not be certain if they were exposed to both, where the information was learned. So, each participant was assigned to either the letter or the Web site condition. That is, the first participant was assigned to the letter condition, the next participant was assigned to the Web condition, then the letter condition, and so forth until all sessions were complete. It should be noted that the letter was displayed on the computer screen, rather than on paper, to enable evaluation via eye tracking.

2.2 Participants

Sixteen participants (8 interacted with the simulated production version of a letter and the other 8 interacted with the Web site) were recruited from Census Bureau’s Business Register, the universe listing for Census Bureau establishment surveys. All were local to the Washington, DC area. (One of the Web participants refused to be audio or video recorded so we were unable to collect any eye-tracking data during his session. In addition, one of the letter participants’ eyes were unable to be tracked with the TOBII software). All were fluent in reading and speaking English, had completed at least two years of college education and were within the age range of 34–68 years old. Participants took part in the research study at their place of work in a voluntary capacity; as such, they were not given any monetary incentive by the Census Bureau. See Table 1 for participant characteristics.

Table 1. Mean (and range) demographics by study condition
Full size table

2.3 Tasks

The task for the letter condition asked that participants read the letter on the laptop computer provided by the test administrator. Since the letter was displayed on the computer screen, the participants were informed that it was two pages long and that they would need to scroll to see all of the information. Participants were instructed to “Please read the letter the way you would if you had actually received it at your workplace, and then let me know once you are finished.” Once the participant said they were finished, the test administrator stopped the eye tracking. A sample letter is depicted in Fig. 1.

Fig. 1.
figure 1

Sample letter

Full size image

The task for the Web site condition asked that the participant imagine that their business had been selected to complete a Census Bureau establishment survey on the Web and that they had received these materials (an envelope with a paper that contained login information) in the mailFootnote 1. The participants were then instructed to “Please open the envelope to find information on how to log in to the survey. I’m going to bring up the survey. Please read over the Web site and login to start the survey.” Once the participant clicked on the login button, the task was finished and the eye tracking was stopped. The sample Web site is depicted in Fig. 2.

Fig. 2.
figure 2

Screen shot of login page of Web survey

Full size image

Participants in each condition then engaged in a number of post-task debriefing questions and follow-up activities. The focus of this paper is the association between the eye-movement data and the first post-task debriefing question, a free recall question. The question was to elicit with verbal probing everything the participant had learned from what he/she had just seen. In the letter condition, the question was “Can you tell me about everything you learned from the letter? Anything else? Anything else? Anything else?” For the Web condition, the probe was the same except “Web site” was used in place of “letter.”

2.4 Procedure

During each one-on-one usability session, the participant was informed that the data were being collected for research purposes only. The test administrator asked the participant to read and sign a consent form stating that he/she understood his/her rights and was voluntarily taking part in the study. The test administrator began the recording after the participant signed the consent form. The participant completed a demographic questionnaire and was positioned in front of the laptop so that a brief eye calibration could be administered. After the participant performed the primary task, the test administrator began the debriefing portion of the study. The sessions lasted no longer than 30 min.

2.5 Equipment

The sessions were conducted at the participants’ business establishments on a laptop (Dell Latitude E6540) that had a Tobii X2-60 eye-tracking device attached to the front of the laptop. For the Web version of the study, the participants used Internet browser IE Version 11. For the letter condition, a PDF was loaded into the eye-tracking software and when the test started, the letter appeared on the laptop screen. Analysis of the eye-tracking data was conducted with the Tobii Studio software [6].

2.6 Eye-Tracking Metrics

We assessed participants’ attention to the letter or Web site with eye tracking. For the letter condition, we looked at total fixation duration, which is the total length of time spent fixating within the identified Areas of Interest (AOIs) in seconds to assess how long participants spent reading each part of the letter. We created the AOIs in the letter condition as simply one AOI for each of the paragraphs in the letter. We also looked at the fixation count so that we could get a measure of the number of fixations per character as the paragraph lengths varied and this measure normalizes the paragraph lengths so that we could compare the amount of time spent on each paragraph, relative to the length of the paragraph [7]. For the Web site, we created the AOIs on different chunks of information on the site, including:

  • Top banner that has the Census logo and the name of the survey

  • Burden statement link

  • One paragraph below the burden statement link that describes what the survey does

  • The next paragraph that mentions the authority and confidentiality statement

  • Instructions located just above the login information

  • Login area

  • Example of where to locate the login information in the paper materials

  • Warning message that is required on all Census survey Web sites

  • Footer links

We looked at fixation duration in each AOI to determine where and for how long participants looked at certain areas of the Web site prior to login. See Appendix A and B for the visual of the pre-identified AOIs. For both conditions, we watched each participant’s animated and static gaze plots. We also examined the relationship between what the eye-tracking data showed and what the participants’ answers were to the free recall question. That is, we reviewed what participants mentioned during the free recall and whether there was evidence in the eye-tracking data of their attention to the area in the letter/Web site that contained that information.

3 Results

First, we present the eye-tracking results for the letter condition. Looking at the total fixation duration of participants on different parts of the letter, we found that five out of seven participants did not read the opening greeting of the letter, and four participants did not read the closing salutation from the Director. Four participants read the first paragraph that mentions the authority of the Census Bureau to collect the survey data. The paragraph that contained the survey’s burden estimate had the longest fixation duration, with an average of 8.96 s. This was followed closely by an average of 8.69 s for the third paragraph, which explains that answers will be kept confidential and used only for statistical purposes. We examined the total number of fixations per character, including only the paragraphs where there was a count of at least six participants’ eye movements, and we found that when we account for the paragraph length, the number of fixations per character is between 0.04 to 0.06. However the “Thank you” paragraph at 0.02 fixations per character is lower than the other paragraphs.

The data on the number of fixations per character indicates that participants were spending about the same amount of time on each paragraph of the letter. Therefore we conclude that there appears to be nothing in the burden estimate paragraph or the confidential paragraph that was unduly difficult to read or drew the eyes for a longer than normal amount of time. As mentioned earlier, the one paragraph that participants spent noticeably less time on was the “Thank you” paragraph. However, this is perhaps not so unusual as the paragraph included a telephone number and a URL, which was not readable as distinct words. It is possible that this content may only be read closely if participants were in a situation where they needed to call for assistance or access help.

Finally we reviewed participants’ free recall verbalizations about what they remembered from the letter. Comments included content that indicated they had read and synthesized the salient pieces of information from the letter, including that it was a monthly survey, it would take 7 min to complete, and the answers would be kept confidential. Two participants commented on what they perceived to be contradictory pieces of information in the letter, and these points are indicators of what areas of the letter could be improved (e.g., usability findings on areas of improvement).

To summarize, eye movements, including gaze plots taken from the letter task, indicate that participants were reading the letter. Most participants read over key parts of the letter including the part of the letter that was bolded. Participants appeared to miss or skim over the letterhead and the signature block. As the number of fixations per character data highlight, the “Thank you” paragraph is also skimmed over. See examples of letter condition gaze plots from four different participants in Fig. 3.

Fig. 3.
figure 3

Gaze plots of 4 different participants in letter condition

Full size image

Next we present eye-tracking results from the Web site condition. Analysis of fixation duration within the AOIs highlight that areas of the Web site that were not attended to include the links at the bottom of the screen (none of the participants looked in that area), along with the warning message (only one participant looked at that area of the screen) and the burden statement link (only one participant looked at that area of the screen). What did draw the attention of the participants’ eyes was the username and password fields (participants spent an average of 1.83 s on this field) and to the right of that field, which displayed an example of where the participant would locate their username and password on the mailing materials (2.18 s on average). In addition, a few participants noticed the instruction just above the username and password field (spent on average 0.99 s). Three participants noticed the section of the Web site that begins “Authority and confidentiality,” which explains that the response is voluntary and that the information that is collected will be kept confidential. They spent on average 30 s reviewing this area of the Web site.

The participants’ free recall verbalizations about what they remember about the Web site mostly matched participant behavior of the eye-tracking data. Comments included that participants did not read the text, that they did not pay attention to the Web site as a whole, and that they did not remember much beyond the username and password area of the site. Four participants explicitly said they remembered the username and password areas on the screen. One participant mentioned that she thought the survey was mandatory (it was voluntary), and another participant mentioned that it was about retail stores and remembered reading about Census Bureau’s authority (this information was contained on the Web site).

Looking at the eye movements and the free recall feedback together, we notice that participants did not spend time looking over the entire Web site but rather focused quickly on the username and login portion of the Web site. Gaze plots from participant behavior of the task give a visual perspective on how participants’ were attending primarily to the username and login fields or the middle section of the Web site. See Fig. 4.

Fig. 4.
figure 4

Gaze plots of 4 different participants in the Web site condition

Full size image

Comparing the results of the letter condition to the Web condition, we learned that participants in the letter condition appeared to read over the entire letter while participants in the Web site condition did not read over the entire Web site but instead focused on the area of the Web site where they had to make an action (e.g., begin the survey by logging in). The free recall verbalizations in the Web condition were quite a bit shorter in length and in content than the verbalizations from the letter condition; however for both conditions, the verbalizations mostly matched the eye-tracking data.

4 Limitations

The letter that participants interacted with was not on paper as it would be in real life but rather, in the test setting, the print letter had been saved as an electronic PDF so that it could be loaded onto the computer to make it possible to conduct eye tracking. Consequently, participant behavior may be different if they had received a paper letter at their place of business. The Web portion of the study did not have the letter included and as such is not true to what a participant would experience in real life. While this was intentional (as we didn’t want participant performance to be influenced by materials they may or may not have read or seen in a prior task), it is not typical to the real-life situation, and participant behavior may be different outside of the laboratory setting. However, we feel the task itself is a typical task in that respondents are asked to go online, login, and begin the survey. In addition, limitations include the spatial accuracy of eye tracking. Gaze position calibration was a challenge, and head movement could compromise the recording of eye movements.

5 Conclusion

In general, we found that participants in the letter condition exhibited indications of reading the letter. Their eyes moved over key points of the letter. During the free recall component of the study we have indications that participants remembered some key points of the letter. On the other hand, other parts of the letter, while read, confused some participants. The participants’ confusion was manifested in debriefing but not in the eye-tracking analysis, which indicates that eye-tracking alone is not always sufficient to understand participant interactions with letter materials.

For the Web site, we found that the task of reading over the Web site was ignored for the act of getting started with the survey. The eye-tracking behavior matches what participants said they recalled about the site: that they were focusing on logging into the survey.

Finally, this pilot study demonstrates that there is some association between the eye-tracking data and the debriefing feedback. Integration of eye-tracking data with other behavioral data appears a promising approach to usability evaluation. Further methodological development is warranted.