User-Centered Evaluation of Visual Analytics

1. Front Matter

   Pages i-xi

   PDF

2. Introduction

   • Jean Scholtz

   Pages 1-2

3. Analysis

   • Jean Scholtz

   Pages 3-5

4. Analytic Methods

   • Jean Scholtz

   Pages 7-10

5. What is Visual Analytics and Why is it Needed?

   • Jean Scholtz

   Pages 11-15

6. User-Centered Evaluation

   • Jean Scholtz

   Pages 17-26

7. Evaluation Needs for Visual Analytics

   • Jean Scholtz

   Pages 27-45

8. Current Examples of Evaluation of Visual Analytics Systems

   • Jean Scholtz

   Pages 47-49

9. Trends in Visual Analytics Research and Development

   • Jean Scholtz

   Pages 51-53

10. Conclusions

    • Jean Scholtz

    Pages 55-56

11. Back Matter

    Pages 57-71

    PDF

Visual analytics has come a long way since its inception in 2005. The amount of data in the world today has increased significantly and experts in many domains are struggling to make sense of their data. Visual analytics is helping them conduct their analyses. While software developers have worked for many years to develop software that helps users do their tasks, this task is becoming more and more onerous, as understanding the needs and data used by expert users requires more than some simple usability testing during the development process. The need for a user-centered evaluation process was envisioned in Illuminating the Path, the seminal work on visual analytics by James Thomas and Kristin Cook in 2005. We have learned over the intervening years that not only will user-centered evaluation help software developers to turn out products that have more utility, the evaluation efforts can also help point out the direction for future research efforts.

This book describes the efforts that go into analysis, including critical thinking, sensemaking, and various analytics techniques learned from the intelligence community. Support for these components is needed in order to provide the most utility for the expert users. There are a good number of techniques for evaluating software that hasbeen developed within the human-computer interaction (HCI) community. While some of these techniques can be used as is, others require modifications. These too are described in the book. An essential point to stress is that the users of the domains for which visual analytics tools are being designed need to be involved in the process. The work they do and the obstacles in their current processes need to be understood in order to determine both the types of evaluations needed and the metrics to use in these evaluations. At this point in time, very few published efforts describe more than informal evaluations. The purpose of this book is to help readers understand the need for moreuser-centered evaluations to drive both better-designed products and to define areas for future research. Hopefully readers will view this work as an exciting and creative effort and will join the community involved in these efforts.

• Pacific Northwest National Laboratory (PNNL), USA

  Jean Scholtz

Jean Scholtz is a chief scientist at the Pacific Northwest National Laboratory (PNNL). Dr. Scholtz came to PNNL in 2006 to work on the Visual Analytics Science and Technology (VAST) Challenge for Jim Thomas, as part of the National Visualization and Analytics Center (NVAC). Although she no longer works on the VAST Challenge, she continues to advocate and conduct user-centered evaluations for various programs at the laboratory. Dr. Scholtz previously worked at the National Institute of Standards and Technology (NIST), where she worked in user-centered evaluations for an Intelligence Advanced Research Projects Agency (IARPA) program, Novel Intelligence from Massive Data (NIMD), and collaborated with PNNL to start up the VAST Challenge. At NIST she was a founder of the Common Industry Format (CIF) project that established a format for software companies to describe their usability testing results. This document is used in requests for information today by companies interested in purchasing software and has become an ISO standard. Dr. Scholtz also worked on human-robot interaction evaluations while at NIST, helping with both Urban Search and Rescue (USAR) and Explosive Ordinance Disposal (EOD) testbeds. She was a program manager at the Defense Advanced Research Projects Agency (DAPRA) involved in collaboration software, digital libraries, and ubiquitous computing. In an earlier career, Dr. Scholtz worked at Bell Telephone Laboratory in Murray Hill, NJ, where she helped develop an early time-sharing system, Project Mac, with researchers from MIT. She also worked on early efforts at missile defense on Kwajalein, MI. Dr. Scholtz has a Ph.D. in computer science from the University of Nebraska at Lincoln, a Master's degree in mathematics from the Steven Institute of Technology in Hoboken, NJ, and a Bachelor's degree in mathematics from the University of Iowa. Dr. Scholtz has been a member of ACM since 1988, has served on a number of CHI conference committees, and was a member of the SIGCHI Board. She received the CHI lifetime service award in 2014. She served for a number of years on the NASA Human Factors in Space Review Board.

Policies and ethics