Insight into the relative merits of rating and ranking in a cross-national context using three-way correspondence analysis

Abstract

In sensory and marketing research both ratings and rankings are frequently used to measure values or preferences. There is no consensus as to the preferred method. Moreover, cross-national studies concerning this topic are scarce, preventing insight into the comparability of measurement procedures across countries. In this paper, we study rating and ranking of values by the same respondents in five countries in the European Union. Insight is given into the relative merits of rating and ranking measurement by using three-way correspondence analysis. Our findings show that rating and ranking are assessed quite similarly by people from different countries. The minor differences that exist may be explained by cultural differences in response patterns in ratings. Finally, we recommend a combined rating–ranking approach to determine the preference order for all values in a set, including those considered to be of moderate importance.

Introduction

Measurement is of key importance in comparative international research. Both ratings (Schwartz, 1992) and rankings (Kamakura and Mazzon, 1991, Rodrigue et al., 2000) are often used for measuring an individual’s preferences. In particular, when dealing with values there is an ongoing debate about the choice for either rating or ranking for its measurement. Ovadia (2004) refers to the current situation as a rating–ranking impasse.

In situations where respondents are asked to indicate preferences or priorities, ranking is recommended by some (Falahee and MacRae, 1997, Kamakura and Novak, 1992, Rokeach, 1973), while others recommend rating (e.g. Schwartz, 1992, Villanueva et al., 2005). However, respondent discrimination between products (Park et al., 2007, Rodrigue et al., 2000) or values (Kamakura and Mazzon, 1991, Ovadia, 2004) is the common aim of all these studies.

In this study we focus on the measurement of values. Values serve as standards to guide the selection or evaluation of behaviour (Schwartz, 1992) and can, because of their generality, be useful in cross-national research. In studies on food preference, personal values are used to create segments (Chryssohoidis & Krystallis, 2005) or are employed in means-end chain studies (Gutman, 1982). Means-end chains is an established method for providing insight into consumers’ motivations in food choice (Costa et al., 2007, Nielsen et al., 1998). They make it possible to relate the attributes of various products, via functional and social consequences, to personal values.

Most studies on rating versus ranking have been carried out in only one country (Russell and Gray, 1994, Villanueva et al., 2005). However, it is known that individuals from different countries may exhibit a different use of scales. In fact, several cross-national studies have found that there are consistent differences in the utilization of the various points on rating scales between countries (e.g. Baumgartner and Steenkamp, 2001, Cox et al., 2001, Van Herk et al., 2004, Yeh et al., 1998).

In this paper, we add to the existing empirical findings on values and preference measurement, by studying rating and ranking in a cross-national setting. Our data set, which encompasses almost 5000 respondents from five European countries, provides a basis for empirical generalizations. Such generalizations are considered to be “a building block of science” (Bass & Wind, 1995). As existing studies on rating versus ranking have been carried out in a single country, it is not known whether research findings from one country can be generalized to other countries. This study provides insight into the cross-national applicability of rating and ranking procedures. Moreover, with this study we respond to the call for further research on this subject made by Ovadia (2004), who suggested comparing a full ranking of values with a rating of the same values.

The contribution of this study is three-fold. First, we provide insight into the usage of rating and ranking procedures in a cross-national setting. Second, in contrast to most previous studies, we employ a within-subject design, which allows us to assess the similarities and differences in rating and ranking scale use by the same individuals. Third, we introduce a methodological approach, three-way correspondence analysis, to the field of international sensory research. This technique was developed in psychology research (Carlier & Kroonenberg, 1996) and it is particularly suited to our analysis as it allows us to simultaneously visualize the relationship between rating and ranking across countries.

This paper is organized in the following way. First, we discuss previous research on rating and ranking. Next, we present the methodological framework of three-way correspondence analysis to simultaneously analyse ratings and rankings across countries. In an empirical study based on the List of Values (LOV; Kahle, 1983), we apply the proposed methodology to obtain insight into the relative merits of rating and ranking in an international setting. We conclude with a discussion, and directions for future research.