Replications of forecasting research

doi:10.1016/j.ijforecast.2009.09.003

International Journal of Forecasting

Volume 26, Issue 1, January–March 2010, Pages 4-8

https://doi.org/10.1016/j.ijforecast.2009.09.003 Get rights and content

Abstract

We have examined the frequency of replications published in the two leading forecasting journals, the International Journal of Forecasting (IJF) and the Journal of Forecasting (JoF). Replications in the IJF and JoF between 1996 and 2008 comprised 8.4% of the empirical papers. Various other areas of management science have values ranging from 2.2% in the Journal of Marketing Research to 18.1% in the American Economic Review. We also found that 35.3% of the replications in forecasting journals provided full support for the findings of the initial study, 45.1% provided partial support, and 19.6% provided no support. Given the importance of replications, we recommend various steps to encourage replications, such as requiring a full disclosure of the methods and data used for all published papers, and inviting researchers to replicate specific important papers.

Introduction

Gardner and Diaz-Saiz (2008) replicated and extended research performed by Fildes, Hibon, Makridakis, and Meade (1998), which was itself an extension of the M-Competition study (Makridakis et al., 1982). Using changes in the estimation procedures, they concluded that the primary conclusion was supported but disagreed with one of the secondary conclusions. This demonstrates the value of replications, both in showing where we can gain confidence and in indicating areas which are in need of further research.

Experts have claimed that replication is vital to scientific progress (e.g. Hunter, 2001). Replications help to ensure that findings can be reproduced. Extensions go beyond that, and examine whether the findings can be generalized.

However, despite these benefits, relatively few of the papers published in various areas of management science are replications (Hubbard and Vetter, 1996, Evanschitzky et al., 2007). A number of reasons for this have been suggested. First, and perhaps most important, many studies in management science are unimportant, and thus it would be senseless to replicate them. Second, authors seldom provide sufficient detail in the paper (or in response to requests) to allow for replication. And, third, reviewers seem to be biased against replications, either because they think that they do not offer anything new or because the results are not statistically significant.

The misinterpretation of null hypothesis testing procedures may also have undermined the perceived need for replication. Oakes (1986) showed that 42 out of 70 (60%) experienced academic psychologists falsely believed that an experimental outcome that is significant at the 0.01 level has a 0.99 probability of being statistically significant if the study were replicated. This is not true, and shows the low level of knowledge among some academics, which is possibly leading to the erroneous conclusion that replications are not needed.

Based on these observations, we examine the state of replication research in forecasting and then suggest ways to make further improvements with respect to replication.

Section snippets

The record of replications in the leading forecasting journals

The definitions of the central terms in this study are extensions of those employed by Hubbard and Armstrong (1994, p. 236). A replication is defined as “a duplication of a previously published empirical study that is concerned with assessing whether similar findings can be obtained upon repeating the study”. Likewise, a replication with an extension is “a duplication of a previously published empirical research project that serves to investigate the ability to generalize earlier research

Discussion

Given the relatively small number of replications across all disciplines of management science, in conjunction with the majority of replication attempts failing to provide support for the initial findings, we call for an increase in published replications of important papers. To aid in this increase, the data and methods used in the original studies should be made available on the Internet, concurrently with a paper’s publication. This is important because over time authors often lose track of

Conclusions

In comparison with the empirical studies published in other areas of management sciences, the replication rate in the International Journal of Forecasting (10.4% of empirical papers published) is well above the median of 6.6%, while the Journal of Forecasting replication rate (6.3%) is slightly below the median. It is difficult to say what the optimum number of replication studies that should be published is. However, given that many replications do not support the original findings (about 1/5

Acknowledgments

We thank all those authors who replied to our request to ensure that they were accurately cited. In addition, Nils Petter Gleditsch, Raymond Hubbard, Bruce D. McCullough and the reviewers provided useful suggestions.

References (14)

H. Evanschitzky et al.
Replication research’s disturbing trend
Journal of Business Research
(2007)
R. Fildes et al.
Generalizing about univariate forecasting methods: Further empirical evidence
International Journal of Forecasting
(1998)
E.S. Gardner et al.
Exponential smoothing in the telecommunications data
International Journal of Forecasting
(2008)
R. Hubbard et al.
Replications and extensions in marketing—rarely published but quite contrary
International Journal of Research in Marketing
(1994)
R. Hubbard et al.
An empirical comparison of published replication research in accounting, economics, finance, management, and marketing
Journal of Business Research
(1996)
W.G. Dewald et al.
Replication in empirical economics: The Journal of Money, Credit and Banking project
American Economic Review
(1986)
N.P. Gleditsch et al.
Posting your data: Will you be scooped or will you be famous?
International Studies Perspectives
(2003)

There are more references available in the full text version of this article.

Cited by (32)

Business and management research: Low instances of replication studies and a lack of author independence in replications
2022, Research Policy
Increasing awareness of the credibility crisis and related replication crisis in business research drives calls for greater understanding of the state of replication studies. This research addresses these calls by analyzing the 10-year publication history of 121 leading journals (Academic Journal Guide (AJG 2018)). Examination of 83,682 articles reveals 4,412 potential replications. Detailed analysis of 500 randomly selected articles offers further insights. Results indicate most replications are conceptual in nature, support prior findings and represent only 1.47% of published research in leading journals. Significantly, most replications occur as part of within-study or intrastudy designs. Replications by independent researchers are very rare, raising credibility concerns due to author overlap and associated researcher and measurement biases. Recommendations for the improvement of replication efforts are made.
Reproducibility and replicability crisis: How management compares to psychology and economics – A systematic review of literature
2021, European Management Journal
The past decade has been marked by concerns regarding the replicability and reproducibility of published research in the social sciences. Publicized failures to replicate landmark studies, along with high-profile cases of research fraud, have led scholars to reconsider the trustworthiness of both findings and institutionalized research practices. This paper considers two questions: (1) Relative to psychology and economics, what is the state of replication and reproduction research in management? (2) Are the disciplines equally advanced in the use of methods applied to study the replication problem? A systematic literature review identified 67 studies pertinent to these questions. The results indicate that the replication prevalence rate in management studies lies almost exactly between those of psychology and economics, while a high level of variation between management and other business-related disciplines can be noted. Further, similarly to psychology, but unlike economics, the surveys of published replications tend to report high replication success rates for management and other business-related disciplines. However, a comparison with recently obtained results in preregistered multi-study replications in psychology and economics suggests that these rates are almost certainly inflated. Method and data transparency are medium to low, often rendering attempts to reproduce or replicate studies impossible. Finally, the understanding of the replicability problem in management is held back by the underutilization of methods developed in other disciplines. The review also reveals that management, psychology, and economics exhibit strikingly different practices and approaches to replication, despite facing similar incentive structures. Disciplines in which replication and reproduction attempts are rare and which frequently involve authors of the original study in replication attempts lack strong deterrents against questionable research practices; thus, they are less likely to deliver replicable results.
Changing research culture toward more use of replication research: a narrative review of barriers and strategies
2021, Journal of Clinical Epidemiology
Citation Excerpt :
Editors should also be trained to have the required skills to review replications and recognize their importance for knowledge development [12,15,66]. Some authors suggested that journals could appoint a replication editor or an editorial board devoted to replication [7,42,66]. Still others stated that it is also important to promote use of adequate reporting guidelines and submission of detailed appendices including experimental protocol, instructions to subjects, measurement tools, and so on [28,49,52,78].
The aim of this paper is to review the literature on barriers to conducting replication research and strategies to increase its use and promotion by researchers, editors, and funders.
This review was part of a larger meta-narrative review aimed at conducting a concept analysis of replication and developing a replication research framework. A combination of systematic and snowball search strategies was used to identify relevant literature in multiple research fields. Data were coded and analyzed using the Theoretical Domains Framework for barriers to replication and the behavior change wheel for solutions.
In total, 153 papers were included in this narrative review. Multiple barriers limit the use of replication research by researchers, editors, and funders. Many of the barriers were related to knowledge and skills of all these actors. Social influences and the research environmental context were also described as not supportive. Multiple strategies were proposed to create positive outcomes expectations, reinforcement, and structural changes in the physical and social context of research.
A social change involving advisory groups, research organizations, and institutions is required to establish new norms that will value, promote, support, and reward replication research.
Replication Research Series-Paper 1 : A concept analysis and meta-narrative review established a comprehensive theoretical definition of replication research to improve its use
2021, Journal of Clinical Epidemiology
Citation Excerpt :
Replication can also refer to a study that only has a certain level of similarity with the original experiment. In this case, researchers purposively make minor or important changes to the study to evaluate possible generalization and extension of previous research findings [1,2,12,31,33,34,35,43,46,47,53–58]. Examples would be changes in either the manipulated or measured variables, investigating the influence of additional variables, repeating the study using different populations, contexts, geographical areas, time periods, or using any combination of the aforementioned changes [59].
The aim of this study is to clarify the concept of replication research to improve its appropriate use by researchers, editors, research funders, and decision makers.
We combined concept analysis and metanarrative review methods to synthetize knowledge on replication research from various scientific fields. We used multiple search strategies to identify the relevant literature published before April 2018. We summarized the data by seeking commonalities and differences in underlying conceptual and theoretical assumptions in the literature.
A total of 153 articles from various disciplines were included. The analysis led to the identification of three major definitions of replication: the repetition of a previous study, the extension of a previous study, and the road-testing of a theory. Attributes, conditions required to conduct replication studies, concerns related to the interpretation of replication studies, and diverse replication research typologies were synthesized, combined, and analyzed. Based on this metanarrative review, a comprehensive theoretical definition of replication research was formulated.
This study can support the adoption of a shared understanding and recognition of the indispensable nature of replication research for the sound development of knowledge in all research fields.
A comparison of brand loyalty between on the go and take-home consumption purchases
2020, Journal of Retailing and Consumer Services
Citation Excerpt :
The primary direction for future research is replication of the findings, preferably in another country, and with other or a greater variety of product categories. Prominent researchers have lamented the lack of replication in marketing for decades (Hubbard and Armstrong, 1994; Evanschitzky and Armstrong, 2010; Uncles, 2011; Lehmann and Bengart, 2016). Hopefully this research area is one in which replications can be conducted.
This paper compares consumer brand purchase loyalty for food products bought either ‘on the go’ (OTG), or for take-home consumption. The study uses two UK consumer packaged goods datasets. The first dataset comprises consumers' purchasing of brands in three product categories: soft drinks, crisps, and savory snacks for on the go consumption. The second contains consumers' purchasing of the same brands for take-home consumption. Analysis uses the polarization index as a behavioral loyalty measure, estimated from the Beta Binomial – Negative Binomial Distribution. This measure controls for the difference in purchase rate and brand market share across on the go and take-home. The study finds that consumer loyalty to brands is markedly higher in purchasing for on the go consumption than for take-home consumption; and that the effect is even stronger for larger brands in on the go.
Skating on thin evidence: Implications for public policy
2018, European Journal of Political Economy
Good public policy needs to be evidence based. However, the evidence base is thin for many policy issues. How can policy makers best respond to such thin areas of research that are also quite likely to change over time? Our survey investigates the evolution of the econometric evidence base for 101 economic issues, drawing upon 42,578 effect sizes (mainly elasticities and correlations) from 4300 econometric studies. We evaluate the performance of six approaches to early research assessment: the simple unweighted mean; the median; the Paldam, “divide by 2” rule of thumb; the unrestricted weighted least squares (WLS) weighted average; the PET-PEESE meta-regression correction for publication bias; the weighted average of the adequately powered (WAAP); and WAAP-WLS. Lowest prediction errors are found in the Paldam rule of thumb and WLS. WLS typically reduces the initial exaggeration of thin evidence by half.

View all citing articles on Scopus

^☆: Invited paper.

View full text

Replications of forecasting research☆

Abstract

Introduction

Section snippets

The record of replications in the leading forecasting journals

Discussion

Conclusions

Acknowledgments

Journal of Business Research

International Journal of Forecasting

International Journal of Forecasting

International Journal of Research in Marketing

Journal of Business Research

Replication in empirical economics: The Journal of Money, Credit and Banking project

American Economic Review

Posting your data: Will you be scooped or will you be famous?

International Studies Perspectives