Based on a comprehensive literature search, this meta review analyzes to what extent past literature reviews on IS business value have covered key research areas and preserved their key findings. The results show that while some areas have been explored extensively, some other crucial areas, such as accounting performance, the growth of intangible assets, and the differentiation between economic output and derived or perceived value, have been neglected. They need to be considered in future reviews. The results also reveal those research areas where even primary research is weak and needs to get intensified before literature reviews can be applied to synthesize findings.

1 Introduction

Information systems (IS) started to be embedded in economic environments many decades ago and are even considered commodity inputs nowadays (Carr 2003). The reliance on IS has meanwhile occurred to an extent that for some firms the failure of IS impedes or even renders business activities impossible. IS have also gained macroeconomic importance: according to the World Information Technology Services Alliance (WITSA 2008, p. 1), the global marketplace for information and communication technology is likely to have topped $3.7 trillion in 2008. The economic relevance of IS has made research on “IS business value” highly attractive to researchers, who have shaped the academic discussion by publishing an abundance of research papers, according to the literature reviews analyzed in this paper.

Some researchers provide sobering arguments on the economic relevance of IS. For example, West and Courtney (1993) and Hitt and Brynjolfsson (1996) doubt the strategic power of IS, and argue that IS are commodities and that any IS-based advantages will be soon eroded. Carr (2003) sums up doubts by even entitling his paper “IT doesn’t matter”. Another discourse is rooted in empirical studies that do not find evidence that IS positively affect performance (Dos Santos et al. 1993; Rai et al. 1997; Im et al. 2001; Dehning and Stratopoulos 2002; Ko and Osei-Bryson 2004; Stiroh and Botsch 2007). Apparently, IS researchers have (at least not fully) managed to identify and to explain the economic relevance of IS. Business executives and researchers continue to question the value of IS investments, as Kohli and Grover (2008, p. 23) note in their recent review. However, answering this question is regarded fundamental to the contribution of the IS discipline (Agarwal and Lucas 2005).

A straightforward approach to reveal IS business value is to synthesize empirical findings of the literature. However, the large number of studies is accompanied by a variety of methods, research objects, research models, and findings. The discussion of IS business value has reached a high level of complexity, which makes it extremely difficult to overlook key research findings. This complexity has been addressed by researchers who published literature reviews in as many as 15 different outlets, including such pertinent journals as MISQ, ISR, JMIS, EJIS, CACM, JAIS, and ACM Computing Surveys. In the presence of the aforementioned critics on IS, the question arises to what extent knowledge on IS business value has been preserved through prior reviews. This leads us to the research question of this paper:

To what extent have past reviews addressed or neglected key areas in IS business value research?

The importance of this question is leveraged by the argument of Kohli and Grover (2008) who hypothesize that past research on IS business value has either disregarded or underemphasized increasingly important research areas and questions.

The overall goal of this work is to answer the research question by (a) identifying and describing central findings in key research areas of IS business value research, and (b) synthesizing what literature reviews have done to preserve knowledge. Through the methodological lens, this paper is a review of literature reviews, and thus a “meta review”. Thereby, it differs from a recently published review on the value of information systems (Urbach et al. 2009) in the research method and in the objects under investigation.

The remainder of this paper is organized as follows: Sect. 2 provides the theoretical background of IS business value research and literature review methodology. In Sect. 3, the research framework and the methodology of this paper are presented. Section 4 uses a taxonomy to condense main fields in IS business value research. Section 5 analyzes to what extent the research fields have been addressed in literature reviews. Finally, Sect. 6 concludes this article and presents specific high-priority recommendations for future research.

2 Theoretical Background

From a methodological point of view, a meta (literature) review is a particular type of review and can thereby draw on review methodology. This section draws on this methodology and follows the recommendation of Webster and Watson (2002, p. xv), who suggest that a review paper should provide elaborate definitions of key variables of the review and should set the boundaries on the review. In this paper, key variables are “information systems” and “IS business value”.

2.1 Information Systems (IS)

The academic field of IS is terminologically pervaded by the usage of syntactically similar notions, such as “information system (IS)”, “information technology (IT)” and “information and communication technology (ICT)”. However, these notions often lack any precise semantic definitions. Reviewing articles published in “Information Systems Research”, Orlikowski and Iacono (2001) find that the “IT artifact” has not been theorized and is widely interpreted depending on the specific research context. The notional fuzziness and heterogeneous semantics in literature is not surprising, because information systems discipline does not yet provide a broadly-accepted or even standardized ontology. In this review, we adopt the “holistic” view on IS, as described in the ATIS Telecom Glossary (ATIS 2007) (option 3): “The entire infrastructure, organization, personnel, and components for the collection, processing, storage, transmission, display, dissemination, and disposition of information.”

2.2 IS Business Value

We frame IS business value research by defining notion and scope and the level, object and time of evaluation.

IS literature offers a variety of notions and semantics. For example, early works use the notions “value”, “benefit”, “outcome” or “worth” (Wiseman 1992), Melville et al. (2004) investigate “organizational performance”, and Kohli and Grover (2008) refer to value as the “economic impact”. This variety in terminology does not only mirror notional inconsistencies, it also reflects different understandings of how to operationalize the economic impact of IS. For example, a large subset of empirical studies apply econometric approaches by analyzing the relationship between IS investments and economic variables, such as productivity (Brynjolfsson and Hitt 1996), “Return on Sales” (Bharadwaj 2000), or Tobin’s q (Brynjolfsson and Yang 1999). Other studies stress that, beyond financial and non-financial measures, intangible assets can be affected by IS investments (Irani 2002; Kohli and Grover 2008). The discussion becomes even more complicated when researchers also distinguish between what the particular outcome of an IS investment is and how this outcome is interpreted. The interpretation of a particular outcome depends on the view of the particular evaluator (Sylla and Wen 2002, 242), on what competitors have achieved (Dehning and Richardson 2002, 23), and what is finally done to exploit it (Alshawi et al. 2003, p. 419). As this review is dedicated to the identification of uncharted territories in IS business value research, it does not exclude any of the aforementioned facets. Rather, they are used to structure research findings.

Literature suggests different levels for the examination of the economic impact of IS. A widely used classification distinguishes individual level, firm level, industry level and economy level (Bakos 1987; Kauffman and Weill 1989; Brynjolfsson and Yang 1996; Devaraj and Kohli 2000; Chau et al. 2007). In addition, research also analyzes consumer surplus (Bakos 1987; Brynjolfsson and Yang 1996; Devaraj and Kohli 2000). This work does not exclude any of these levels.

Consistent with the holistic definition of IS adopted in this paper, we address the economic impact of investments in information technology, in organizational assets, and in personnel.

As Kohli and Grover (Kohli and Grover 2008, p. 25) stress, research on IS value can be of “ex ante” and “ex post” nature. While “ex ante” research is closely related to decision making, “ex post” research is dedicated to the control of past expenses. This work includes both streams of research.

3 Research Design and Methodology

The methodology used in this paper is based on the theoretical research framework shown in Fig.  1 . The bold rectangles and arrows indicate those parts that are focused in this work.

Fig. 1
figure 1

Methodological research framework

Full size image

“Literature review” is an established research methodology (Salipante et al. 1982; Cooper and Hedges 1994; White 1994). It is of particular importance for IS research, as stressed by Webster and Watson (2002, p. xiii f), who argue that the literature review “[…] facilitates theory development, closes areas where a plethora of research exists, and uncovers areas where research is needed. […][T]he literature review represents the foundation for research in IS. As such, review articles are critical to strengthening IS as a field of study.” The relevance of literature reviews has also been addressed in renowned IS journals. For example, several years ago “MIS Quarterly” launched its “MISQ Review Department” (Watson 2001), a unit dedicated to the publication of literature reviews. Another example is the journal “WIRTSCHAFTSINFORMATIK”, which publishes this literature review in its “State-of-the-Art” column. The journals “European Journal of Information System” and the “Journal of Management Information Systems” are examples of renowned journals that explicitly include review papers and surveys in their scope of invited contributions.

Apparently, literature reviews are an appreciated and highly important methodology in IS research. This paper draws on this importance twofold: it analyzes those reviews that address the domain “IS business value”, and it applies (meta) review methodology by itself.

Meta review methodology is still in its infancy in terms of methodology and application. However, a straightforward approach is to apply “review methodology”, which can be regarded as a methodological generalization of “meta review methodology”. Thus, we apply “review methodology” and draw upon the work of Webster and Watson (2002). They particularly stress the importance of identifying relevant literature and structuring the review.

We performed a title search in pertinent journal databases, namely Business Source Premier, MLA International Bibliography, EconLit, ScienceDirect, IEEE Xplore, the ACM Digital Library, and Web of Science. The logical search string was (“information technology” OR “information systems”) AND (“value” OR “investment” OR “productivity” OR “competitive” OR “performance” OR “measurement” OR “evaluation” OR “profit” OR “efficiency”). We further scanned the table of contents of the following journals (listed in alphabetical order): Academy of Management Review, ACM Transactions on Information Systems, American Economic Review, Communications of the ACM, European Journal of Information Systems, Information Systems Journal, Information Systems Research, Journal of Management Information Systems, Journal of the AIS, Management Science, MIS Quarterly, and Wirtschaftsinformatik.

Regarding the structure of the review, we apply a concept-centric approach, with research fields being the concepts. More specifically, we adapt the matrix approach of Salipante et al. (1982) by tabulating review articles against research areas (concepts).

This article considers 22 literature reviews on IS business value research, which have been published since 1989 in peer-reviewed journals or peer-reviewed conference proceedings. More specifically, we use the following reviews, which are listed in chronological order and described in detail in the appendix: (Kauffman and Weill 1989; DeLone and McLean 1992; Brynjolfsson 1993; Soh and Markus 1995; Brynjolfsson and Yang 1996; Potthof 1998; Sircar et al. 1998; Seddon et al. 1999; Bannister and Remenyi 2000; Chan 2000; Devaraj and Kohli 2000; Dehning and Richardson 2002; Irani and Love 2002; Sylla and Wen 2002; Dedrick et al. 2003; Melville et al. 2004; Walter and Spitta 2004; Piccoli and Ives 2005; Chau et al. 2007; Wan et al. 2007; Kohli and Grover 2008; Pare et al. 2008).

In order to avoid confusion between “research papers” and “literature reviews”, it should be noticed that we use the findings of research papers to define key research fields in Sect. 4. Literature reviews are used in Sect. 5 in order to analyze the extent to which these fields have been covered in literature reviews.

4 Key Areas of IS Business Value Research

The literature on IS business value provides a variety of taxonomies, which are rooted in different perspectives of the authors. For example, DeLone and McLean (1992) analyze the dependent variable and suggest categories of IS success, Seddon et al. (1999) provide a taxonomy that accounts for the type of IS asset used and different stakeholders, and Irani and Love (2002) focus on IS investment evaluation methodology and provide a taxonomy of investment appraisal techniques. As the goal of this paper is to provide a broad picture of concepts in IS business value research, we do not focus on a single perspective or taxonomy. We rather identify those dimensions that are widely adopted in the literature. Finally we use these dimensions to shape the taxonomy on which our meta literature review is based upon.

We find broad consensus in the literature that important dimensions of IS business value are “performance measure” (DeLone and McLean 1992; Barua et al. 1995; Dehning and Richardson 2002; Melville et al. 2004; Chau et al. 2007), the “level of measurement” (Bakos 1987; Brynjolfsson 1993; Dehning and Richardson 2002; Pare et al. 2008), the “type of IS asset” (Weill 1992; Mahmood and Mann 1993; Rai et al. 1997; Seddon et al. 1999; Sircar et al. 2000; Melville et al. 2004), “methods” (Chan 2000; Irani and Love 2002; Chau et al. 2007; Pare et al. 2008), and “influencing factors” (contextual factors, lag effects, risk) (Weill and Olson 1989; Barua et al. 1995; Davern and Kauffman 2000; Stiroh 2002; Ko and Osei-Bryson 2004; Melville et al. 2004; Dewan et al. 2007). While these dimensions address the measurement of IS performance, researchers also stress the importance of questioning what the value of a particular performance is (Dehning and Richardson 2002; Alshawi et al. 2003). Thus, we add the dimension “value” to our taxonomy, which is shown in Fig.  2 .

Fig. 2
figure 2

Taxonomy of IS business value research

Full size image

We describe each of these dimensions in the following subsections and derive key research fields. Prior to applying this procedure, we explain first why we consider “terminology” an additional research field.

4.1 Terminology

For each academic discipline, a consistent terminology is essential to name relevant constructs, to define its semantics and to resolve potential ambiguities. However, the discussion in Sect. 2 already revealed some confusion in IS literature. The importance of clearly defining the subject of research is pinpointed by Orlikowski and Iacono (2001, p. 121): “[…] we propose that IS researchers begin to theorize specifically about IT artefacts, and then incorporate these theories explicitly into their studies.” We define “Research field 1: Terminology”.

4.2 Performance Measure

Researchers have analyzed a variety of economic measures, such as productivity (Brynjolfsson and Hitt 1996, 2000), production efficiency (Thatcher and Oliver 2001), consumer welfare (Thatcher and Pingry 2004), profit ratios (Weill 1992; Barua et al. 1995), and also market-oriented measures (Bharadwaj et al. 1999; Brynjolfsson and Yang 1999). The abundance of different aspects of IS success is addressed by researchers who provide taxonomies to organize the diverse research (DeLone and McLean 1992; Irani and Love 2002; Gable et al. 2008). A simple and often applied classification distinguishes between process performance and firm performance, which subsumes market performance and accounting performance (Barua et al. 1995; Dehning and Richardson 2002; Melville et al. 2004). It is widely agreed that the impact of IS investments on firm performance is intermediated by process performance (Barua et al. 1995; Soh and Markus 1995; Dehning and Richardson 2002; Kim et al. 2006; Mittal and Nault 2009).

Among process performance measures, productivity is most intensively discussed. Some early studies in the late 1980s and early 1990s did not find that IS considerably contributed to productivity and economic growth at economy level (Baily 1986; Roach 1987; Jorgenson and Stiroh 1995), at industry level (Roach 1991; Berndt and Morrison 1995), or at firm level (Loveman 1994). One impact of these studies was the creation of the term “productivity paradoxon”. However, with IS becoming a larger share of total capital investment (Dedrick et al. 2003, p. 19), more recent studies find a major impact of IS investments on productivity and economic growth in developed countries (Jorgenson and Stiroh 2000; Oliner and Sichel 2000; Jorgenson 2001). At firm level, the picture seems to be less clear: While some studies (Ko and Bryson 2002; Ko and Osei-Bryson 2004; Lin and Shao 2006b) do not find any evidence of a positive correlation or suggest a microeconomic explanation (Stickel 1995), opposite results are reported by Brynjolfsson and Hitt (1996, 2000), Kelley (1994), Lin and Shao (2006a), Neirotti and Paolucci (2007), Menon et al. (2000), Stiroh (2002), and Swierczek and Shrestna (2003). This leads to the definition of “Research field 2: Productivity”.

Researchers have shown their interest to analyze to what extent IS investments are correlated with increased (stock) market performance of firms. Tam (1998) and Brynjolfsson and Hitt (1996) investigate the impact on “Total Shareholder Return”, Dos Santos et al. (1993) and Im et al. (2001) analyze stock market reactions, and Bharadwaj et al. (1999) and Brynjolfsson and Yang (1999) focus on Tobin’s q. Although some studies find a positive correlation, Dedrick et al. (2003, p. 10) argue that this correlation is of purely temporal nature, but lacks any causal characteristics, as many more micro- and macro-economic factors determine market performance. On the other hand, Brynjolfsson and Yang (1999) and Brynjolfsson et al. (2002) suggest that adjustment costs and intangible assets may provide an explanation for the high market valuation found for IS. Bharadwaj et al. (2009) adopt the opposite perspective by analyzing the effects of information technology failures on the market value of firms. Their results reveal that the market responds negatively to IS failures. To conclude, we define “Research field 3: Market performance”.

The impact of IS investments on accounting performance in terms of cost ratios, turnover ratios and profit ratios is one the most intensively studied research areas in IS business value research. Cost ratios are analyzed by Bharadaj (2000) and Santhanam and Hartano (2003). Turnover ratios are investigated in the studies of Dehning and Stratopoulos (2002) and Barua (1995). Many studies address profit ratios: IS investments seem to positively affect “Return on Sales” (Tam 1998; Bharadwaj 2000; Dehning and Stratopoulos 2002; Santhanam and Hartono 2003) and “Operating income to employees” (Bharadwaj 2000; Santhanam and Hartono 2003), while the positive impact on “Return on Assets” (Hitt and Brynjolfsson 1996; Rai et al. 1997; Tam 1998; Bharadwaj 2000; Stratopoulos and Dehning 2000; Dehning and Stratopoulos 2002; Santhanam and Hartono 2003), “Return on Investment” (Stratopoulos and Dehning 2000; Hayes et al. 2001; Mahmood and Mann 2005), and “Return on Equity” (Alpar and Kim 1990; Rai et al. 1997; Tam 1998; Stratopoulos and Dehning 2000) is less clear. We define “Research field 4: Accounting performance”.

While the aforementioned performance measures address tangible benefits, the importance of intangible benefits, such as increased capabilities and knowledge at organizational level, or better decision making, has often been acknowledged (Mertens et al. 1982; Soh and Markus 1995; Brynjolfsson and Hitt 2000; Irani and Love 2001) and was recently re-emphasized by Kohli and Grover (2008). Bhatt and Grover (2005) even argue that the quality of IS business expertise can form capabilities that have a significant effect on competitive advantage. However, only few research papers address intangible benefits. To sum up, we define “Research field 5: Intangible benefits”.

4.3 Level of Measurement

Literature suggests different levels for the examination of the economic impact of IS (see Sect. 2.2). Several studies limit their investigations to a particular level. For example, Brynjolfsson and Hitt (1996, 2000) and Mahmood and Mann (2005) focus on firm level, Shih et al. (2007) adopt a macro-economic view at country-level, and Devaraj and Kohli (2000), Brynjolfsson (1996) and Hitt and Brynjolfsson (1996) analyze consumer surplus created by IS investments. The importance of taking the level of examination into account is stressed by Dehning and Richardson (2002, p. 8) and by Brynjolfsson (1993), who states that the usage of different levels even contributes to the explanation of the productivity paradoxon. Apparently, the separation of different levels is useful to structure research and to resolve allegedly conflicting results. But it is also argued that, beyond the separation of levels, their linkage can provide useful insights and explanations of how IS generates value (DeLone and McLean 1992; Kohli and Grover 2008). We define “Research field 6: Level of measurement”.

4.4 Type of IS Asset

It has been widely argued in the literature that better insights in the way IS investments induce superior business performance require a breakdown of IS investments into single IS assets (Weill 1992; Mahmood and Mann 1993; Rai et al. 1997; Sircar et al. 2000; Melville et al. 2004). IT capital-related studies (Hitt and Brynjolfsson 1994; Barua et al. 1995; Rai et al. 1997; Tam 1998; Sircar et al. 2000; Mahmood and Mann 2005) find no correlation with stock market behavior, mixed results regarding profitability ratios, and a positive correlation with profitability in terms of “sales” and “value added”. Some studies (Kelley 1994; Rai et al. 1997) are even more specialized and analyze the impact of hardware expenditures or expenditures based on investments in software (Rai et al. 1997), production-oriented software (Barua et al. 1995), interorganizational information systems (Schumann 1990), ERP systems (Poston and Grabski 2000; Hayes et al. 2001; Karimi et al. 2007), e-commerce systems (Subramani and Walden 2001), supply chain systems (Kim et al. 2006), knowledge management systems (Maier and Hädrich 2001) or infrastructure (Rai et al. 1997; Byrd and Turner 2000; Chatterjee et al. 2002). The studies differ enormously in methods, data, time period, and indicators used. This conclusion also applies to studies that are related to IS personnel and training expenditures (Sircar et al. 2000; Chatterjee et al. 2001; Mahmood and Mann 2005). We define “Research field 7: Type of IS asset”.

4.5 Methods

Studies of decision practice indicate that managers often avail themselves of relatively simplistic cost-benefit analysis in the context of traditional capital budgeting (Bannister and Remenyi 2000; Irani and Love 2002; Chau et al. 2007). However, beyond traditional capital budgeting, many more approaches have been proposed, such as those related to measuring accounting or market-based measures (see the discussion above). The portfolio of proposed methods also includes value analysis (Money et al. 1988) and analysis based on critical success factors (CSF) (Rockart 1979). Overall, the literature on performance measurement provides a plethora of different appraisal methods (Bannister and Remenyi 2000, p. 232). To sum up, we identify “Research field 8: Methods”.

4.6 Influencing Factors

It is widely argued in the literature that the impact of IS investments on economic performance is influenced by non-technological factors. Mostly discussed are factors related to economic structures (contextual factors), lag effects, or risk. We briefly discuss each of them.

Contextual factors comprise firm, industry, and economic factors. They have been found to affect the economic impact of IS investments (Weill 1992; Bharadwaj 2000; Davern and Kauffman 2000; Dehning and Richardson 2002; Ko and Osei-Bryson 2004; Melville et al. 2004; Zhu et al. 2004). Most studies focus on firm factors (Floyd and Wooldridge 1990; Li and Ye 1999; Ravichandran and Lertwongsatien 2005; Chari et al. 2008). These studies strongly suggest that (a) the alignment of IS with a firm’s core competencies and business planning and (b) close ties between IS investments and upper management are crucial for IS-driven enhanced firm performance. Competitive factors are addressed in the works of Lin and Shao (2006b), Sircar et al. (2000), and Melville et al. (2007), macro-environmental factors are analyzed in the contributions of Swierczek and Shrestha (2003) and Zhu et al. (2004). We define “Research field 9: Contextual factors”.

It is argued in the literature that a mismeasurement of IS investment impact may be rooted in inappropriate methodology, when delayed effects need to be considered, but are ignored (Weill and Olson 1989; Stiroh 2002). Some empirical studies (Santhanam and Hartono 2003; Mahmood and Mann 2005) account for this criticism and find that lags may exist and that several years may pass before an organization’s investment in IT bears fruit. We consider this phenomenon by defining “Research field 10: Lag effects”.

As in the case of many other investments, IS investments bear economic risks due to the uncertainty of future and states (McFarlan 1981; Wehrmann et al. 2006). IS investments are regarded even substantially riskier than non-IS investments, as measured by their relative contributions to the overall riskiness of the firm (Dewan et al. 2007, p. 1829). The (ex ante) evaluation of IS investments is also based on personal expectations and risk preferences of decision makers (Rose et al. 2004, p. 53). Risk in IS investment decisions is explicitly considered in the papers of Au and Kauffman (2003), Wehrmann and Zimmermann (2005), Wehrmann et al. (2006), Benaroch et al. (2007), and Dewan et al. (2007). As risk is deemed a substantial component of IS investment decisions, we define “Research field 11: Risk”.

4.7 Value

While the economic performance of IS investments is usually determined by measuring and comparing economic ratios, some researchers started questioning what the value of a particular outcome is. It is argued that the actual value of an outcome may depend on what is done with newly generated capabilities (Alshawi et al. 2003, p. 419), what competitors have achieved (Dehning and Richardson 2002, p. 23), and what the subjective preferences of the persons who perform the evaluation are (Sylla and Wen 2002, p. 242).

The distinction between what is measured and how this outcome is finally valued has already been substantiated in decision theory and utility theory, which distinguish between the result of measurement (referred to as “outcome”) and the perceived value.

One of the most intensively discussed types of IS value is competitive advantage. It is argued that competitive advantage can only be gained if firms apply strategic information management (Zahn 1990), and if IS-based capabilities are pretended from being imitated by competitors (Feeny and Ives 1990; Carr 2003) or if competitors do not fully benefit from imitation (Clemons and Row 1991). West and Courtney (1993, p. 245) note that any advantage of innovation will be eroded as the technology becomes common practice. Hitt and Brynjolfsson (1996) find that, although IS investments do not lead to competitive advantage, they are necessary to maintain competitive parity. An even more positive picture is drawn by Bhatt and Grover (2005), who find evidence in their empirical study that the quality of IT business expertise and the relationship infrastructure have significant effect on competitive advantage. Fink and Neumann (2009) show that IS personnel knowledge and skills positively affect the range of managerial IS infrastructure capabilities, which in turn are responsible for perceived competitive impacts. To sum up, the value of IS represents an important research question. It shapes “Research field 12: Value”.

5 Analysis

This section analyzes to what extent the research areas identified in the previous section have been addressed in literature reviews. Table  1 provides an overview of the results.

Table 1 Coverage of “IS business value” research areas in literature reviews
Full size table

5.1 Terminology

Two reviews (Kauffman and Weill 1989; Melville et al. 2004) briefly investigate literature regarding terminology. The work of Kauffman and Weill (1989), which reveals inconsistent definitions of input and output variables, embraces a very early period in IS business value research. However, 15 years later we are informed by Melville et al. (2004) that the IS community has still divergent perspectives on the IT construct, which depend on the specific context of research. Although the work of Melville et al. (2004) provides only a brief overview of terminology and perspectives, it is an excellent starting point for future literature reviews.

5.2 Productivity

The large interest of researchers in exploring the impact of IS investments on productivity is also mirrored in the number of literature reviews that address productivity. DeLone and McLean (1992) provide an early overview on productivity studies. A comprehensive review is conducted by Brynjolfsson (1993), who conclude that the alleged productivity paradoxon is much due to deficiencies in measurement and methodology, more precisely in mismeasurement of inputs and outputs, lags due to learning and adjustment, redistribution and dissipation of profits, and mismanagement of information and technology. A further deficiency is identified by Sircar et al. (1998), who conclude that many studies that claim to inspect productivity rather measure firm performance. Interestingly, Sircar et al. (1998) also find that the underlying theory impacts results: while studies based on variance theory refute the productivity paradoxon, those based on process theory support it.

Brynjolfsson and Yang (1996) prefer to conduct productivity research at firm-level because this helps to control many problems from aggregation occurring at industry level. Overall, they find a positive effect on productivity reported in recent literature. This conclusion is supported in the more recent reviews of Devaraj and Kohli (2000) and Dedrick et al. (2003), who also admit that the impact varies widely among different companies. According to Wan et al. (2007), the productivity paradoxon has been resolved at firm level due to more sophisticated and refined data sources, a shift in the level of analysis towards the firm level, and a refocus on the management of IS. They argue that research has probably better accounted for the four problems cited by Brynjolfsson (1993). At the industry level, results are less clear. Devaraj and Kohli (2000) find mixed results in the literature, and Dedrick et al. (2003) identify some positive returns in the form of labor productivity. Reviewing productivity at economy level, early studies failed to identify positive effects of IS investments. However, in the 1990s more positive results occurred (Brynjolfsson and Yang 1996) and seven years later Dedrick et al. (2003) find that literature has shown a positive relationship between IS investments, growth and national productivity, at least in developed countries.

Literature reviews on productivity have provided excellent overviews of productivity at different levels and have synthesized the findings of research papers regarding the question of whether IS investments led to increased productivity or not. However, this perspective does not allow explaining why the impact differs so much and resolving the conflicting results of studies that found positive results and those that did not.

5.3 Market Performance

Literature provides some studies that find a positive correlation of IS investments and market performance and that IS have a mediated impact (Brynjolfsson and Yang 1999; Brynjolfsson and Hitt 2000). Although there is only few literature available, which makes “market performance” less attractive for literature reviews, two early reviews (DeLone and McLean 1992; Dehning and Richardson 2002) analyze literature findings. The review of DeLone and McLean (1992) identifies few empirical studies only. The review of Dehning and Richardson (2002, p. 19) concludes that market values increase by 5 to 20 times the amount spent on IS and that shareholders value strategic IS investments. However, since 2003 the interest in the investigation of the impact of IS investments on market performance has declined and there is almost no recent research papers to get reviewed.

5.4 Accounting Performance

Interestingly, the abundance of empirical studies on accounting ratios has been addressed in detail by two literature reviews only. While DeLone and McLean (1992) find too few studies to draw an overall picture, Dehning and Richardson (2002) find that the relation between IS spending and accounting performance is tenuous. However, in contrast to market performance, accounting performance continues to attract researchers’ interest (see, for example, the study of Mahmood and Mann 2005).

5.5 Intangible Benefits

Although intangible benefits have been addressed in research papers only rarely, several literature reviews acknowledge that the benefit of IS investments encloses intangibles (DeLone and McLean 1992; Soh and Markus 1995; Devaraj and Kohli 2000) and that IS is an enabler of organizational changes that can lead to additional productivity gains (Dedrick et al. 2003). The review of Sylla and Wen (2002) suggest to apply techniques related to multi-objective and multi-criteria analysis, value analysis and critical success factors. Kohli and Grover (2008, p. 33) state that our measurement instruments are often too blunt to capture intangibles.

5.6 Level of Measurement

The classification of Bakos (1987) is widely adopted in literature reviews. For example, Brynjolfsson and Yang (1996) and Dedrick et al. (2003) use firm, industry, and economy level to analyze literature findings on productivity (see Sect. 5.2). Brynjolfsson and Yang (1996) argue in favor of firm level, as going down to this level helps to control many problems from aggregation.

The reviews of Chan (2000), Chau et al. (2007) and Wan et al. (2007) reveal that the firm level has attracted most of researchers’ interest in the past (about 80% of all studies investigated). Their results also show that very few studies combine multiple-level approaches and that research at the individual level has been particularly underemphasized. A more balanced picture is drawn by Pare et al. (2008), who find that 19% of empirical studies focus on group level, 23% on individual level, and 26% on firm level. Multiple levels are addressed in 14% of the investigated studies. However, the comparison of results of the aforementioned studies is difficult, as they refer to different outlets and periods.

5.7 Type of IS Asset

The impact of specific IS assets (or combinations) has not attracted much attention in review literature, although there are many research papers available and although the impact of particular IS assets and combinations in conjunction with their use and contextual factors is highly relevant for IS investment decision makers. The review of Seddon et al. (1999) is a valuable exception. It uses the type of IS asset to classify IS effectiveness literature. However, the authors do not classify and assess literature according to the particular IS asset investigated.

5.8 Methods

The diversity in methods has been recognized and addressed in some literature reviews. As early as 1989, Kauffman and Weill analyzed applied methods and found that the majority of studies are exploratory and mostly based on microeconomic theory. Potthof (1998) finds that many empirical studies show deficiencies in terms of data and/or methods used. These deficiencies weaken the significance of the overall positive results. Chan (2000) find that the period 1993–1998 was methodologically predominated by secondary data and market data analyses, and case studies. Analyzing a more comprehensive period (1991–2005), but also limiting their analysis to four leading IS journals, Pare et al. (2008, p. 407) find that experiments, case studies and questionnaire surveys account for 74% of all research papers. Schumann (1993), Irani and Love (2002), and Walter and Spitta (2004) provide taxonomies for evaluation techniques. More recently, Chau et al. (2007) analyzed ECIS (2000–2005) and PACIS (1993–2005) papers and found a general shift from using objective measures (firm value, ROI) to perceptual measures. Overall, the reviews on research methods provide a good exploratory overview of this research field. However, only few publications tell us when to use which method. Exceptions are the works of Walter and Spitta (2004), and Sylla and Wen (2002) who survey methods and propose a conceptual framework that helps decision makers to choose the most appropriate method. The authors discuss various evaluation techniques for tangible benefits, intangible benefits, and risk.

5.9 Contextual Factors

The role of contextual factors to determine the impact of IS investments is widely discussed in literature and results have also been reviewed. Dehning and Richardson (2002) identify the particular role of contextual factors for abnormal stock market returns. Dedrick et al. (2003) highlight the importance of organizational capital, such as decentralized decision-making systems, job training, and business process redesign. Melville et al. (2004) stress that the organizational and technological context impacts magnitude and type of operational efficiencies. Ravichandran et al. (2009) find that the interaction between IS spending and product and geographical diversification can have a positive effect on firm performance. At industry level, Melville et al. (2004) find that the degree of competition in an industry correlates positively with the extent to which firms achieve efficiency gains, but negatively with the extent to which firms are able to capture the benefits of efficiency gains. At the macro environmental level, they identify the telecommunications infrastructure as important factor for the economic value of interorganizational information systems.

5.10 Lag Effects

The need to take lag effects into account was already stated by Kauffman and Weill (1989), who concluded in their review that time lags are often omitted from models. Some years later, Brynjolfsson (1993) and Brynjolfsson and Yang (1996) even argued in their reviews that lags due to learning and adjustment have been insufficiently considered in productivity studies and that this shortcoming in methodology is one of four explanations of the “IT productivity paradoxon”. These reviews provide an excellent analysis of the impact of lag effects on productivity. Unfortunately, literature findings on the relevance of lag effects in research fields other than productivity have been neglected in past literature reviews.

5.11 Risk

Risk in the context of IS investments has received little attention in research papers and does not provide a fertile area for reviews. “[The] consideration of risk is virtually absent in the growing literature on the returns on IT investment, even though the risks are widely recognized.” (Dewan et al. 2007). However, one review (Sylla and Wen 2002) addresses risk and describes briefly the application of real option, portfolio approach, and Delphi approach in the context of IS risk.

5.12 Value

Researchers have started to allude to the difference between the economic outcome and the value that is perceived or derived. Although none of the analyzed reviews systematically addresses literature findings on IS value, three reviews address the competitive advantage induced by IS. Melville et al. (2004) find that the degree to which the firm can obtain a sustained competitive advantage is determined through the level of inimitability of rare organizational resources that are complementary to IT, and through lacking substitutes. Kohli and Grover (2008) stress that leveraging IS and complementarities can lead to competition-strengthening “differential value”. The review of Piccoli and Ives (2005) provides an excellent synthesis of work that examines the role of IS in sustaining competitive advantage. According to this review, the literature has coalesced around four determinants of sustainability of IT-dependent strategic initiatives: 1. IT resources barrier (IT assets and IT capabilities), 2. complementary resources barrier, such as organizational structure, governance, or access to distribution channels, 3. IT project barrier (technology characteristics and implementation process), and 4. preemption barrier (switching costs and value system structural characteristics).

6 Conclusion

Based on a comprehensive literature search, this meta review analyzes to what extent past literature reviews on IS business value have covered key research areas and preserved their key findings. The results show that while some areas have been explored extensively, some other crucial areas have been neglected and should be considered in future research. The results also reveal research areas where even primary research is weak and needs to get intensified before literature reviews can be applied to synthesize findings. More precisely, the main results of this paper are as follows:

First, the research fields “level of measurement” and “contextual factors” have been addressed comprehensively in reviews.

Second, there are research fields where large parts have been covered effectively in reviews, but where some subfields have been neglected. The field “productivity” has been addressed extensively in exploratory reviews. With the exception of two early reviews (Brynjolfsson 1993; Brynjolfsson and Yang 1996), research still lacks explanatory reviews that identify complementary assets and relevant contextual factors. Similarly, literature reviews have covered the research field “methods” comprehensively in terms of which methods have been used (exploratory perspective). However, only little work has been done to analyze the appropriateness of various techniques. With regard to the research field “lag effects”, past reviews have effectively synthesized literature findings that refer to productivity, but they have not gone beyond productivity. While the role of IS in gaining sustainable competitive advantage has been considered well in literature reviews, a more general perspective on the subtle difference between economic performance and business value is desirable. The aforementioned research fields provide fertile areas for future literature reviews.

Third, we find research fields (“terminology” and “intangible benefits”), which have not been extensively investigated in research papers. However, some literature reviews stress the importance of the fields and provide excellent starting points for future reviews, which would be, in turn, good starting points for primary research.

Fourth, there are fertile research areas that have been (largely) ignored by reviews (“accounting performance” and “type of IS asset”). These fields should be addressed with high priority.

Fifth, there are important research fields (“market performance” and “risk”) where no substantial body of research was available for literature reviews. We suggest that researchers (re)start covering these fields.