A small world on feet of clay? A comparison of empirical small-world studies against best-practice criteria
Introduction
Around 1958, a manuscript of political scientist Ithiel de Sola Pool and mathematician Manfred Kochen circulated among network researchers (Deutsch, 1989, Kochen, 1989). In this manuscript, the two scholars set out to explain a familiar experience, now commonly referred to as the “small-world” experience: the experience of two strangers meeting at a place far away from home and finding out to share a common acquaintance. For the two anecdotal protagonists the world feels small because a serendipitous event like the small-world experience runs against their intuition about the likelihood of such an event (Kochen, 1989, p. vii).
But how likely is it objectively that two randomly chosen individuals do share a common acquaintance, given the structure of acquaintanceship in society? And to ask a more general version of this question: “[What is] the distribution of length of minimum contact chains: how many pairs of persons in the population can be joined by a single common acquaintance, how many by a chain of two persons, how many by a chain of three, etc.?” (Pool and Kochen, 1978, p. 17). In short, Pool and Kochen planned to examine the distribution of social distances, more specifically the geodesic paths, between pairs of actors, an endeavor deemed relevant by them with regard to potential processes to occur in these paths, e.g., certain kinds of political communication and individual attempts at gaining access to influential circles (Pool and Kochen, 1978, Kochen, 1989, p. vii).
At a time when large-scale online networks were not yet heard of, and most empirical network studies were restricted to data from small and local networks, Pool and Kochen's strategy for an answer was theoretical. They developed a formal model of a large-scale acquaintance network in order to explain theoretically how short chains of acquaintanceship might be possible between any two randomly chosen actors, and how these chains would be distributed according to their length. The work of Pool and Kochen can be seen as part of a larger tradition of formal network modeling, together with other attempts thereof, e.g., the theory of random-biased nets that was originally developed by Rapoport and colleagues (cited in Watts, 1999, p. 12f), and the modeling of small-world networks (Watts and Strogatz, 1998, Newman, 2003, Newman, 2000).
A different and empirical approach to the small-world phenomenon was taken by Milgram and his students. They invented an experimental method to directly measure chains of acquaintanceship and thereby approximate social distance between actors, or categories of actors, in a population. For the first experiment, the researchers drew a sample of starters and asked each starter to direct a letter to one preselected target person, a Boston stockbroker, on the basis of very limited information about the target person. If the starters knew the target person on a first-name basis, they were asked to send the letter to that target person directly. If not, they were asked to send the letter to an acquaintance that they believed to be more likely to know the target. The recipient of the letter was then asked to do the same, i.e., to pass on the letter to the target person either directly or indirectly. Thereby letter chains were started, each consisting of a starter (S), a variable number of intermediaries (I), and a target person (T) (Fig. 1) (for details see: Milgram, 1967, Milgram, 1969, Travers and Milgram, 1969, Korte and Milgram, 1970, Bernard and Killworth, 1979, Kleinfeld, 2002). The first experiment yielded an average of 6.4 steps from the starter to the target person (Travers and Milgram, 1969),1 thus providing the empirical estimate that later led to the famous phrase “six degrees of separation” (Guare, 1990, Watts, 1999, p. 4).
There are a number of empirical and methodological flaws in Milgram's studies and similar letter referral studies, casting doubt on the validity of the method as a measure of network structure, in particular, mean geodesic paths and the distribution of geodesic paths in a population. Among these problems are sample selection biases, chain censoring, small sample sizes of either starters and/or targets, and non-optimal chain-referral strategies (Bernard and Killworth, 1979, Erickson, 1979, Kleinfeld, 2002, Killworth et al., 2006). Given these problems and the current, wide availability of data on complete large-scale networks resulting from electronic traces of processes, one could argue, that chain-referral experiments might have passed their time. The publication trend over time with regard to relevant studies seems to further confirm this assumption: of 14 small-world studies examined for this paper, 10 were published between 1967 and 1979 (Milgram, 1967, Korte and Milgram, 1970, Lundberg, 1975, Erickson and Kringas, 1975, Bochner and Orr, 1979, Bochner et al., 1976a, Bochner et al., 1976b, Guiot, 1976, Shotland, 1976, Lin et al., 1977), and the four remaining studies in 1983, 1991, 1997, and 2003 (Weimann, 1983, Stevenson and Gilly, 1991, Stevenson et al., 1997, Dodds et al., 2003a).
Despite the current unpopularity of letter referral experiments, there are a number of important research contexts in which variants of this approach might still be usefully applied in future research. The potential of letter referral studies lies in (1) approximating structural parameters where other methods are not available, (2) studying the determinants of success and failure for naturally occurring referral processes and the quality of their outcomes, (3) comparing different network structures to make inferences about the kind of processes that can likely occur in these networks. By clarifying the status of referral processes in small-world networks, these empirical studies would thus help to better integrate knowledge about structures and processes (cf. Schnettler, 2009), and it would provide individual and organizational actors with important recommendations on how to best obtain certain resources through their direct and indirect chains of acquaintances.
- 1.
Focus on structure: While more and more data on complete online networks become available, these remain selective in terms of the population they describe: first, access to the Internet is still selective (see Lenhart et al., 2003), and second, not every online social network represents complete acquaintanceship networks. A network of instant messages, for instance, leaves out communication via other channels with the same persons and excludes those persons completely who do not use instant messaging at all. Even social networking websites like Facebook or MySpace, designed very flexibly regarding potential usage, do not capture all of the interaction of its members, and record only part of their social networks (cf., e.g., Lenhart and Madden, 2007). Thus, in those areas of social life where no affiliation and/or process data are accumulated electronically, and no complete network data are sampled with other methods, small-world studies might, despite their biases, still offer one of the best estimates of social distance available. One has to be aware, though, that people are very inaccurate in making the next referral choice: in a comparison of actual geodesics and chains based on peoples’ referral choices, the latter were about 40–50% longer than the former (Killworth et al., 2006). Referral chains are thus only very rough approximations of geodesic paths, and the small-world method should only be used if no better measurement method is available. It is likely that referral methods will play a further decreasing role in the future, as people chose to increasingly interact in virtual worlds, offering new data possibilities for researchers (see Bainbridge, 2007).
- 2.
Focus on processes: Knowledge about chains of acquaintanceship is only meaningful if we know about the processes that occur in such indirect channels (cf. Schnettler, 2009). Thus, another important application for experimental referral chains is the study of factors that affect referral success in a variety of social processes, and that affect the maximum length of chains up to which these processes remain active. On one level, these would be factors influencing peoples’ willingness to cooperate in each step of the chain, and on a second level these would be factors influencing their ability to make strategically optimal choices concerning subsequent referrals. Variants of the small-world experiment allow to systematically vary characteristics of the individuals (e.g., education), or of the referral process itself (e.g., use of incentives). And, as a further extension of this potential research program, it would also be interesting and important to study how different factors affect the efficiency of the referral process, or the quality of the outcome of the referral process in comparison to other, more formal means (cf. McDonald and Elder, 2006). Such a research focus on social processes has been neglected in the area of small-world research (cf. Schnettler, 2009), with a few notable exceptions, e.g., the research on respondent-driven sampling (Heckathorn, 1997, Heckathorn, 2002, Salganik and Heckathorn, 2004) and on job matching through social contacts (Granovetter, 1973, Granovetter, 1974, Granovetter, 1983; critically, cf. McDonald and Elder, 2006). Research on the factors of success and failure of referral processes would provide important insights for individuals, businesses, and other organizations who want to strategically use their direct and indirect networks. But the promise of this research focus depends on the creativity of researchers in modeling new referral experiments that successfully imitate naturally occurring social processes. One example is a study in which starters seeded complaint-referrals in a hospital that were then traced on their way to the appropriate problem handlers (Gilly et al., 1991).
- 3.
Probing structures for their resource potential: A third potential application is at the intersection of structure and processes. Even though not much is known about targeted referral processes where a starter seeks resources, information, or access through indirect chains of acquaintanceship (see Schnettler, 2009), there is at least some evidence that many such processes might be limited to chains much shorter than the average six degrees of separation, e.g., when it comes to seeking access to political representatives (Erickson and Kringas, 1975) or to finding relevant job information (Granovetter, 1974). Only under very specific and rare conditions, e.g., when no other means to obtain resources, information, or access are available, and/or incentives for referral intermediaries are offered, might referral processes sustain over six or more degrees of separation, e.g., in respondent-driven sampling (Heckathorn, 1997) or the search for an abortionist (Lee, 1969). Given such limits on referral processes, a comparative structural analysis might shed light on the potential resources of different groups, and thus importantly contribute to the sociological study of inequality. Letter referral studies could be applied to compare the social distance of certain social groups, e.g., countries, organizations, or occupational groups, in a way similar to previous research on distance between racial groups (Korte and Milgram, 1970) or status groups at a university (Shotland, 1976). Another example of such an application would be the probing for “micro-worlds” or “elite” networks where members are connected through very short chains, and where these short chains imply potential access to a vast variety of resources. In comparative experimental studies of this sort, low accuracy of people in making referral choices (see Killworth et al., 2006) is less problematic if it can be assumed that in the compared structures people are similarly inefficient in making their referral choices.
Given these potential uses of future letter experiments, it seems important to carefully examine the methodological and empirical problems that have plagued past referral studies, and to learn from these mistakes how future studies could be improved. This is the task of the current paper. To this purpose, I will first provide a detailed account of best-practice criteria for referral studies. In a next step, I will briefly introduce a number of small-world studies, that I will then evaluate on the basis of this recommended best practice. The overview shows that hardly any of the existing studies comes up to this standard.
Section snippets
Best-practice criteria for referral experiments
In order to provide guidance for future research applying referral methods, I will sketch a number of important criteria that can serve us to determine the quality of an empirical letter referral study. Such a standard has, to my knowledge, never been explicitly developed as a coherent recommendation, but a few of these criteria have been singled out in previous reviews of small-world research (Erickson, 1979, Bernard and Killworth, 1979, Kleinfeld, 2002).
- C1.
As any empirical study that aims at
Summary of empirical small-world studies
A total of 14 small-world studies, including Milgram's two original studies, could be identified for a comparison in this article. In this section, I will first give a very brief overview of how these studies differ, and in the next Section I will provide a more detailed examination of the empirical and methodological weaknesses of these studies, as well as of some of the measures that have been suggested in the literature to correct for these weaknesses.
In the two original studies of Milgram
Methodological evaluation of small-world studies
In the following I will examine the methodological and empirical problems that many of the referral studies exhibit. To this purpose, the current section is organized according to the best-practice criteria presented in Section 2.
Summary and conclusion
In the introduction of this paper I outline three areas of research in which empirical small-world studies could be usefully applied in the future. I then suggested a number of best-practice criteria for such studies and examined in detail how previous small-world studies compared against this standard. It turns out that none of them completely fulfills all the quality criteria for a best practice. Of those examined, the one that probably comes closest to this standard is the study by Shotland
Acknowledgements
I am deeply indebted to Scott Boorman who accompanied me through all phases of this paper with his helpful and detailed comments. I also thank Silke Aisenbrey, Richard Breen, Hannah Brückner, Anette Fasang, Christine Gockel, Karl Ulrich Mayer, and the participants of the Comparative Research Workshop at Yale University for their useful comments on earlier drafts of this paper. Furthermore, I wish to thank two anonymous reviewers for their helpful suggestions on improving the paper during the
References (58)
- et al.
Social networks: the accuracy of small world chains in social networks
Social Networks
(2006) A structured overview of 50 years of small-world research
Social Networks
(2009)The not-so-small world—ethnicity and acquaintance networks in Israel
Social Networks
(1983)- et al.
Analysis of large-scale social survey
Sample size determination: a review
The Statistician
(1997)The scientific research potential of virtual worlds
Science
(2007)- et al.
A review of small world literature
Sociological Symposium
(1979) - et al.
Communication patterns in an international student dormitory: a modification of the ‘small world’ method
Journal of Applied Social Psychology
(1976) - et al.
Acquaintance links between residents of a high rise building: an application of the ‘small world’ method
The Journal of Social Psychology
(1976) - et al.
Race and academic status as determinants of friendship formation: a field study
International Journal of Psychology
(1979)
Book review of: university communication networks
American Journal of Sociology
The small world problem: the growth of a research idea
An experimental study of search in global social networks
Science
An experimental study of search in global social networks
Science
Some problems of inference from chain data
Sociological Methodology
Small world of politics, or seeking elites from the bottom up
Canadian Research Sociology Quarterly
The flow of crisis information as a probe of work relations
Canadian Journal of Sociology
On small world statistics
Psychometrika
Dynamics of complaint management in the service organization
The Journal of Consumer Affairs
The strength of weak ties: a network theory revisited
Sociological Theory
Getting a Job: A Study of Contacts and Careers
The strength of weak ties
American Journal of Sociology
Six Degrees of Separation
A modification of Milgram's small world method
European Journal of Social Psychology
Respondent-driven sampling II: Deriving valid population estimates from chain-referral samples of hidden populations
Social Problems
Respondent-driven sampling: a new approach to the study of hidden populations
Social Problems
Treating data collected by the ‘small world’ method as a Markov process
Social Forces
Dyadic Data Analysis
Survival Analysis: Techniques for Censored and Truncated Data
Cited by (18)
That'll move the chains: Collecting network chain data
2022, Social NetworksCitation Excerpt :However, except for the comparisons provided in reviews of small world studies (Schnettler, 2009a; Neal et al., 2015a), there is no common language or framework for describing the design features of network chain data collection, which limits the transparency and reproducibility of this work. At the same time, the challenges encountered by these past studies have generated suggestions for best practices in the collection of network chain data (Schnettler, 2009b). However, these best practices tend to be abstract, and have not been operationalized and evaluated in practice, which limits their applied value.
Dynamic Social Networks: Search and Data Routing
2018, Cooperative and Graph Signal Processing: Principles and ApplicationsSurveying data on connected personal networks
2014, Travel Behaviour and SocietyCitation Excerpt :Because they lack knowledge about overall network structure and possible shortcuts, chain lengths are often overestimated (Kleinberg, 2000 for an illustration, see Fig. 1b). Another problem results from questions on how to deal with disrupted contact chains that did not reach their destination (Schnettler, 2009b) (for an illustration see Fig. 1b). In comparison to a small world experiment, snowball sampling asks initial respondents, called “seeds”, to report their social contacts.
A structured overview of 50 years of small-world research
2009, Social NetworksCitation Excerpt :An exception, with an even wider geographic reach than the Milgram studies, is a recent small-world study using starters who self-selected to sign up on the study website and to participate in email referrals to a small number of targets located in different cities worldwide (Dodds et al., 2003a,b). In this study an average of 4.1 steps necessary to link starters and targets is reported, but only 0.4% of all individuals initially willing to participate, and only 1.6% of all starters led to successful chains, a success rate much lower than in most other small-world studies (see Schnettler, 2009 for details; Watts, 2003, p. 135). The average vertex-to-vertex degrees of electronically traced large-scale social networks ranges between about 3 and 16 (Newman, 2003, p. 182).
On the utilization of shortest paths in complex networks
2021, IEEE Access