Introduction

Many countries are making full-fledged efforts to expand renewable energy sources to reduce carbon globally. Paris agreement binds all nations to undertake ambitious efforts to combat climate change (Vicedo-Cabrera et al. 2018). This agreement’s overall goal is to increase the global average temperature to well below 2 °C above pre-industrial levels (Rogelj et al. 2016). The relevant plans for the national units were published as follows. The US provided a mid-century strategy (MCS) for deep decarbonization. This report suggested 80% below 2005 by 2050 (House 2016). EU (2020) presented its long-term strategy for reducing greenhouse gas emissions. Their objective is to achieve a climate-neutral by 2050. Japan submitted its intended nationally determined contribution (INDC). The aim of Japan is ‘decarbonized society’ as soon as possible to 2050 with disruptive innovation focused on Carbon Capture Utilization and Storage (CCUS) and hydrogen energy (Japan 2019). China signed the Paris climate agreement in 2016. It is the world leader in investment in renewable energy and recently declared its aims for ‘carbon neutrality by 2060’ in the UN general assembly. South Korea has also been expanding its policies on renewable energy since the Paris climate agreement. In 2019, the Korean government designated hydrogen energy as the nation’s new growth industry engine and announced a hydrogen economic vitalization roadmap to implement it (Korea 2019). In 2020, to overcome the economic recession caused by the spread of the Covid-19, the government announced a 160 trillion won the Korean New Deal, which focuses on digital and green (Korea 2020).

Korea achieved remarkable economic growth centered on manufacturing, supported by technological innovation (Woo et al. 2014). A manufacturing-oriented increase has also accompanied environmental pollution, making efforts to improve it (Chun et al. 2016). The country’s technological innovation capacity is being used for research and development of renewable energy, and the Korean government is providing a budget for R&D of various renewable energy sources.

Previous performance analysis studies on renewable energy have been conducted on various energy sources, and the main research trends are as follows. In the mid-2000s, DEA had been used in the renewable energy area. After that, in mid-2010, R&D efficiency researches on renewable energy started. Recently, the influencing factors of R&D performance of renewable energy were analyzed by some scholars. Existing prior studies, despite their efforts, have some limitations. First of all, it was difficult to examine the actual performance differences between various energy sources by conducting performance analysis on individual renewable energy sources. From the perspective of government policymakers, it is necessary to make up for the limitations in this area if we recall the comprehensive observation of various energy sources. Second, it was challenging to make decisions based on the situation of R&D due to the lack of distinction between R&D investment, innovation performance, and economic performance. It is necessary to conduct a performance analysis with a different perspective between the two outcomes.

This study aims to respond to the growing importance of renewable energy worldwide and make up for the limitations of existing prior research. This study aims to analyze the performance of R&D projects, including various renewable energy sources, and examine the factors that affect them. We looked at hydrogen, bio, solar photovoltaic, wind power energy sources as renewable energy sources, big firms, universities, research institutions, and small & medium-sized enterprises (SMEs) as R&D organizers. In addition, we consider The R&D stage, which is divided into basic, applied, and development. The performance analysis was conducted for each point of view, separating the economic performance from the innovation performance that was not covered in previous research.

In order to conduct performance analysis, DEA is applied, and input-oriented variable return to scale (VRS) model is used. DEA has demonstrated its reliability within various prior studies. The input variables were government and private R&D investment. The number of papers and patents were outputs as innovation performance, and total economic value and new employment were set as the output of economic performance. Besides, we tried to present meaningful implications by conducting comparative analysis by renewable energy sources, R&D organizer, and The R&D stage based on DEA analysis efficiency score.

Government policymakers need to consider various influencing variables, including renewable energy mix, and this study will provide meaningful information. In addition, it is expected that R&D’s performance will be divided into innovation and economic performance perspectives. Comparative analysis results will be presented for each point of view to help make decisions for each R&D purpose.

This study consists of five chapters. In chapter 2, a literature review was prepared that included DEA methodology and R&D performance analysis of renewable energy. Chapter 3 described the research model and data of this study. Chapter 4 presents detailed DEA results and comparative analysis. Finally, the last chapter concludes that includes a summary of the main findings, implications and limitations of this research, and future research directions.

Materials and methods

Status of renewable energy R&D in Korea

Countries around the world are expected to invest heavily in solar and wind power to foster new industries in the energy sector through energy conversion. The World Energy Organization predicts that solar energy will generate more electricity than coal or gas in 2035. The expansion of renewable energy is expected to be a global trend, and developed countries are also expected to invest aggressively. The current status and prospects of renewable energy generation in Germany, the US, and Japan are likely to increase (Fig. 1).

Fig. 1
figure 1

Installed power generation capacity by source in the Stated Policies Scenario, 2000–2040 (IEA 2020)

Full size image

Korea’s renewable energy R&D accounts for the most considerable portion of the energy and resources budget. The Korean government is expanding R&D investment in renewable energy for the energy transition. From 2016 to 2018, investment in renewable energy was 566.7 billion won, 565.9 billion won, and 871.5 billion won, respectively. The average annual increase rate amounts to 24 percent. The private sector invested twice as much in renewable energy R&D as the government. As of 2018, the private sector has invested 1.7176 trillion won in renewable energy R&D (MOTIE 2020). In 2021, the company plans to expand investment in mass production by the energy source to secure economic growth for renewable energy. It plans to raise support for the development of mass production technology by an energy source such as high-efficiency solar energy, high-capacity wind power generation, and mass production of hydrogen. It also plans to develop infrastructure technologies that can efficiently transfer and utilize the energy produced (MOTIE 2020).

Looking more closely at Korea’s renewable energy R&D status, solar energy, wind power, hydrogen, and new energy materials were selected as the primary technology areas to foster new energy industries. To that end, it invested 251.3 billion won in 2020. In the field of solar energy, it will develop technologies for improving the efficiency of next-generation solar cells and reusing modules. In wind power, it includes the development of floating offshore wind power system and core components localization, and the development of infrastructure technology to develop ultra-large(20MW) wind power. Hydrogen and fuel cells include localization and safety technology of hydrogen charging core components, securing the economy of storage and transportation, etc. It invested 275.5 billion won in 2020 to provide clean and safe energy. In the field of energy safety, demonstration research is carried out to secure safety technology for hydrogen infrastructure and energy convergence facilities and Energy Storage System (ESS) safety. It invested 136.5 billion won in spreading distributed energy. In the distributed energy system field, the development of smart Power Conversion System (PCS) technology and the development of high power and high-reliability ESS technology are included.

DEA models

DEA is a nonparametric methodology used to compare the efficiency of Decision Making Units (DMUs). DEA has several advantages over parametric methodologies. First, DEA can deal with multiple inputs and outputs in a single model (Wang and Huang 2007). Second, DEA allows us to distinguish between efficient and inefficient targets and identify the causes of inefficiency (Wang and Huang 2007). Third, DEA does not need to define a particular function, so it does not need to be a precondition, such as a normal distribution (Guan et al. 2006, Martı́n-Cejas 2002). DEA is an appropriate methodology because various inputs and outputs must be considered at the same time to derive R&D efficiency (Wang and Huang 2007).

Farrell (1957) introduced the concept of efficiency by taking into account several inputs and outputs. Since 1978, various DEA models have been proposed. Charnes, Cooper, and Rhodes suggested the CCR model (Charnes et al. 1978). The CCR model follows the rules of Constant Returns to Scale(CRS), in which all DMU inputs and outputs increase or decrease at the same rate change. Therefore, the CCR model is also called the CRS model. Later, Banker, Charnes, and Cooper developed the BCC model. The BCC model assumes a Variable Returns to Scale (VRS) in which the proportion of inputs and outputs varies depending on economies of scale or scope (Banker et al. 1984). The BCC model is also called the VRS model. The VRS model is divided into IRS, which has a higher output growth rate than the input growth rate, and DRS has a lower output growth rate than the input growth rate (Hollanders and Esser 2007). The efficiency of the BCC model is called pure technical efficiency (PTE), which is divided into technical efficiency (TE) and scale efficiency (SE). PTE and SE can be compared to determine the cause of inefficiency. DEA models are divided into two types depending on input or output orientation. The input-oriented model aims to fix the output elements and minimize the input elements.

In contrast, the output-oriented model seeks to set the input elements and maximize the output elements (Ramanathan 2003). Considering the purpose of the study and the characteristics of each element, the DEA model should be set up and analyzed. Input-oriented and output-oriented models are as follows. In Eqs. (1) and (2), \(\theta\) and \(\phi\) are the efficiencies of the input-oriented and output-oriented model. n is the number of DMUs. k and r represent the number of inputs and outputs. zi is the weight of the model. xi and yr represent the inputs and outputs, repectively. s− and s+ are the slacks of inputs and ouputs, respectively.

$$\begin{array}{*{20}l} {{\text{Min }}\theta } \\ {{\text{Subject to}}} \\ \;\;\;\; {\sum\limits_{{j = 1}}^{n} {z_{{ij}} x_{{ij}} + s^{ - } = \theta x_{0} \quad i = 1, \ldots,k} } \\ \;\;\;\; {\sum\limits_{{j = 1}}^{n} {z_{{rj}} y_{{rj}} - s^{ - } = \theta y_{0} \quad r = 1, \ldots,m} } \\ \;\;\;\;{z_{i} \ge 0,j = 1, \ldots,n} \\ \;\;\;\; {{\text{If VRS}}} \\ \;\;\;\; {{\text{Add}}\;\sum\limits_{{j = 1}}^{n} {z_{j} } = 1} \\ \end{array}$$
(1)
$$\begin{array}{*{20}l} {{\text{Max}}} \\ {{\text{Subject to}}} \\ \;\;\;\; {\sum\limits_{{j = 1}}^{n} {z_{{ij}} } x_{{ij}} + s^{ - } = x_{0},i = 1, \ldots,k} \\ \;\;\;\; {\sum\limits_{{j = 1}}^{n} {z_{{rj}} } x_{{rj}} + s^{ + } = \phi y_{0},r = 1, \ldots,m} \\ \;\;\;\;{z_{j} \ge 0,j = 1, \ldots,n} \\ \;\;\;\; {{\text{Add}}} \\ \;\;\;\;\; {\sum\limits_{{j = 1}}^{n} {z_{j} } = 1} \\ \end{array}$$
(2)

Literature on renewable energy using DEA

As seen in the 2.1 section, the Korean government is expanding R&D investment in renewable energy for the energy transition. However, the efficiency of R&D investment in renewable energy is not high. It is questionable whether it will continue to expand R&D investment for renewable energy during the Long-term low growth era (New Normal). Therefore, it is necessary to investigate the prior research on analyzing the R&D of renewable energy through DEA and seek ways to increase the R&D efficiency of renewable energy in Korea.

In the mid-2000s, DEA began to be used in the renewable energy sector. DMUs were diverse, including countries, companies, and renewable energy technologies. Chien and Hu (2007) analyzed the impact of renewable energy generation on efficiency in 45 Organisation for Economic Co-operation and Development (OECD) member countries and non-entry countries. Labor, capital, and energy consumption were used as inputs, and real Gross Domestic Product (GDP) was used as output. Renewable energy accounted for a large portion of total energy production in non-OECD countries, but the technology efficiency of OECD member countries was higher. Barros (2008) looked at the efficiency changes in 25 hydroelectric power plants in Portugal from 2001 to 2004. Halkos and Tzeremes (2012) analyzed efficiency by utilizing enterprises' financial performance in Greece’s renewable energy sector. Wind energy efficiency was more efficient than hydropower, and solar energy production will be significant in Greece’s renewable energy sector. Unlike previous studies, San Cristóbal (2011) analyzed the efficiency of each renewable energy technology. The inputs included the investment ratio, implementation period, operation and maintenance costs, and the output factors were power generation, operating hours, useful life.

Research on renewable energy R&D efficiency began in mid-2010. Kim et al. (2015) analyzed the R&D efficiency in wind power, solar energy, and fuel cells. The input factors were an investment in technology development and investment in technology diffusion. The output factors were the number of patents, the total volume of power generation, and the unit cost of energy production. In 2007, fuel cells and wind power were efficient energy, but in 2008 and 2011, only wind power was found to be efficient energy. Wang et al. (2016) used two-stage DEA to analyze the R&D efficiency and marketing efficiency of enterprises in China's renewable energy sector. Data were used from 2009 to 2013 for 38 companies listed on the Shanghai and Shenzhen stock exchanges. The inputs were fixed assets, wages, and R&D costs, the R&D output was software assets and revenue, and the marketing phase output used total revenue and market value. Companies in China’s renewable energy sector had relatively lower R&D efficiency than marketing efficiency, and R&D efficiency took the form of an invert u-shape.

Recently, some studies explored the determinants of R&D efficiency of renewable energy. Chang et al. (2020) analyzed the effect of environmental policy on the R&D efficiency of renewable energy enterprises. The government subsidiary and tax refund policies showed a positive impact on total investment efficiency and technology efficiency. The effect of the government subsidy policy was greater than the tax refund. Chun et al. (2016) analyzed the factors affecting hydrogen energy R&D projects. The efficiency of the hydrogen energy R&D projects invested for 10 years was analyzed through DEA method. The type of R&D, the type of technology, and the amount of private investment influenced hydrogen R&D efficiency Table 1.

Table 1 Summary of literature
Full size table

Research design

Model

In order to investigate the performance of renewable energy R&D projects, the authors applied an input-oriented VRS DEA model. DEA needs inputs and outputs variables for measuring relative efficiency. We categorized two perspectives that are innovation and economic performance. Firstly, from an innovation perspective, government investment and private investment are input variables. The number of papers and patents are output variables. Secondly, from an economic perspective, two investment sources are input variables as same as the innovation perspective. In addition, total economic value and the number of new employment are output variables. The total economic value is the sum of revenue from the domestic market, revenue from export, the amount of cost-cutting, the amount of import-substituting effect, and the amount of technology transfer. The relative performance of renewable energy R&D projects is calculated by the ratio of inputs and outputs using DEA.

The Figure 2 is a research flow chart including important steps from data collection to gap analysis. This figure shows the summary of each stage’s main variables and analysis methodologies.

Fig. 2
figure 2

Research flow chart

Full size image

Data

This paper uses data from the Korea institute of Energy Technology Evaluation Planning (KETEP). The KETEP is a government institute for efficient management of government energy R&D programs. The inputs and outputs data are collected and maintained by the KETEP. Specially, all outputs of each R&D project are tracking by first hand. Commonly, R&D performance assumes time-lag duration from inputs to outputs. Because each research’s outputs are not tracked first hand, researchers do not match inputs and outputs within each R&D project. However, in this research, we matched inputs and outputs within each R&D project, and we do not need to assume an appropriate time-lag duration. That is, the strong point of the data, and it may improve the value of the empirical results. The renewable R&D project data is collected from 2010 to 2014 in terms of the finished year. Each R&D project data is tracked for three years after the finished year. For example, if the A project was finished in 2010, this project is monitored from 2011 to 2013. The outputs are calculated by average during three years of tracking duration. Table 2 provides basic information of each variable, including category, description, and unit of measurement (Figs. 3 and 4).

Table 2 Basic information of variables about renewable energy R&D projects
Full size table
Fig. 3
figure 3

DEA model for evaluation of renewable energy R&D project innovation performance

Full size image
Fig. 4
figure 4

DEA model for evaluation of renewable energy R&D project economic performance

Full size image

Results and dicscussion

Ex-post evaluation of the performance of renewable energy R&D projects

Table 3 is the result of descriptive statistics of renewable energy R&D projects. Based on the mean of inputs, we can grasp that the government budget is larger than the private one. In terms of output variables, the number of patents and total economic value are high. The mean score of DEA is different between innovation and economic perspectives. Innovation performance is higher than economic performance based on the mean.

Table 3 Overview of variables used in ex-post evaluation of renewable energy R&D projects
Full size table

Based on Table 3, government’s R&D investment is larger than private’s in terms of mean and standard deviation. In addition, output variables’ standard deviation is larger than mean. It means that DMU’s outputs have a large gap between min and max. It may affect DEA efficiency score. Innovation and economic performance are 0.225 and 0.178, respectively. These are relatively low, and there is a big gap between min and max efficiency scores. These results can be seen as derived from a significant gap input and output variables’ data.

Comparative analysis for investigating influencing factors on performance

The DEA result does not provide any meaningful implications. A comparative analysis is needed, like nonparametric methods with categorizing variables. DEA is a nonparametric methodology, and its’ result also may be nonparametric information. Kruskal-Wallis H test is a widely used methodology for nonparametric data at difference among 3 groups or more. This research applied the Kruskal-Wallis H test to investigate different performances within each influencing factor. In order to conduct a comparative analysis, we categorized R&D projects by renewable energy source, R&D institution, and The R&D stage. Renewable energy source includes hydrogen, bio, solar photovoltaic, and wind power. R&D institution is categorized big firm, university, research institution, and SMEs. Lastly, The R&D stage includes basic, applied, and development. Figs. 5, 6, 7 provide a ratio within each variable. Each figure includes number, and % ratio. It will improve readability.

Fig. 5
figure 5

Number (%) of projects by renewable energy source

Full size image
Fig. 6
figure 6

Number (%) of projects by R&D institution

Full size image
Fig. 7
figure 7

Number (%) of projects by The R&D stage

Full size image

According to Table 4, innovation performance is different from the renewable energy source. The solar photovoltaic efficiency score is higher than hydrogen. This result is supported based on the Kruskal-Wallis H test with statistical significance. Mean ranks, and mean efficiency score is derived from innovation performance DEA score. In the process of applying the Kruskal-Wallis H test, the mean ranks are derived, and the mean efficiency scores among energy source groups were separately analyzed and presented together to increase readability. Tables 5, 6, 7, 8, 9 are also presented using the same way. The underlined values mean that they have statistical significance (Levels of statistical significance: *** = 1%, ** = 5%, and * = 10%).

Table 4 Renewable energy source as an influencing factor on innovation performance
Full size table
Table 5 R&D organizer as an influencing factor on innovation performance
Full size table
Table 6 The R&D stage as an influencing factor on innovation performance
Full size table
Table 7 Renewable energy source as an influencing factor on economic performance
Full size table
Table 8 R&D organizer as an influencing factor on economic performance
Full size table
Table 9 The R&D stage as an influencing factor on economic performance
Full size table

We confirmed solar photovoltaic R&D projects have the highest innovation performance compared to other renewable energy sources.

R&D organizer is also affecting innovation performance. Based on Table 4, if a different R&D organizer performs R&D, their innovation performance might be different. Especially, there is a big innovation performance gap between universities and big firms. This result is supported by statistical significance using the Kruskal–Wallis H test.

Lastly, The R&D stage is an influencing factor in renewable energy R&D project’s innovation performance. Basic R&D project group has the highest DEA score. Especially, there is a big gap between basic and development R&D projects. That may reflect properties of basic R&D that are advantageous for the creation of papers and patents.

This research also investigates economic performance by categorizing variables. First of all, the renewable energy source and performance gap are similar to innovation performance. Like innovation, solar photovoltaic is superior to hydrogen R&D projects. This result is also supported by statistical significance. These results indirectly show that the maturity of solar photovoltaic technology is higher than that of hydrogen energy.

R&D organizer is also affecting economic performance. The main gap exists between big firms and universities. Big firms and universities have the biggest performance gap in terms of economic perspectives. Renewable energy is a kind of state-of-art, and at a new stage of development, if we consider it to be a promising industry sector in the future, we expect this difference in performance to decrease in the future.

The R&D stage does not have any effect in terms of economic performance. It is a different result compared to innovation performance. It implies that in order to create meaningful economic value, the development stage R&D needs to seek a more efficient way. If related markets are expanded to a meaningful size, it will be easier to generate economic value.

Lastly, Table 10 shows the result of the Wilcoxon matched-pairs signed-rank test to understand the difference between the innovation and economic efficiency rank of each R&D project. Like the Kruskal–Wallis H test, the Wilcoxon matched-pairs signed-rank test is a common methodology for nonparametric data at the difference between two statistics based on the same sample. We derived innovation and economic performance based on the same sample through different output variables.

Table 10 Wilcoxon matched-pairs signed-rank test
Full size table

The Wilcoxon matched-pairs signed-rank test was conducted on each sample R&D project based on the efficiency score, both innovation and economic perspectives. The result connotes that renewable energy R&D efficiency is not consistent across projects. In other words, R&D projects experience an imbalance throughout innovation and commercialization. Especially, more than half of the samples to be analyzed showed higher innovation performance than economic performance. In order to improve comprehensive performance, related experts need to consider the way to enhance performance balance.

Conclusion

This paper applied input-oriented VRS DEA model to investigate the renewable energy R&D project performance. The data are from 2010 to 2014, and every 205 R&D project ex-post outputs were tracked three years after the finished year. The authors tried to measure the relative efficiency of R&D projects with innovation and economic perspectives separately. In addition, some influencing factors on performance were investigated through comparative analysis. Lastly, this study analyzed the difference between innovation and economic performance based on each R&D project. The main findings are as follows.

Firstly, overall, innovation and economic performance of renewable energy R&D projects are relatively low. The standard deviation is greater than the average, indicating a significant difference in performance between R&D projects. The result came as the Korean government took a ‘selection and concentration’ strategy to efficiently use its limited renewable energy R&D budget. Despite this strategy, the R&D efficiency of the renewable energy project is not high, so policy decision-makers should consider what factors determine R&D efficiency. Secondly, using comparative analysis, we checked the effect of renewable energy sources, R&D organizers, and the R&D stage on innovation performance. The innovation performance is the difference with statistically significant based on these three factors. R&D efficiencies of solar photovoltaic in renewable sources, university in R&D organizers, and basic R&D are superior to other groups. In terms of economic perspective, there are similar results. Solar photovoltaic is one of the critical technologies in the policy of transiting renewable energy. Following the global trend, the economic and social ripple effects of solar photovoltaic are potentially expected to be significant. Studies show that when a university is a research director for renewable energy projects, it can be assumed that innovation performance leads to economic performance. The efficiency of the projects carried out by companies is low because Korean renewable energy companies have lower cost competitiveness than Chinese renewable energy companies. The basic R&D projects showed relatively high innovation performance to suit the research’s purpose, and the applied and development research did not achieve enough economic performance. It is necessary to establish a life cycle R&D support system in which innovation performance can be linked to economic performance following the purpose of applied and development research. Lastly, Table 10 shows us that innovation performance is higher than economic performance. This test was conducted on each sample R&D project ranking based on DEA score. Renewable energy R&D projects need to improve their success rate of commercialization. In other words, renewable energy R&D projects experienced an imbalance between two perspectives with statistical significance.

This research tried to investigate the overall performance and influencing factors of completed renewable energy R&D projects. This study analyzed various renewable energy sources that were not attempted in previous prior studies. In addition, it is meaningful that the results of innovation and commercialization perspectives were carried out separately. The main findings of this research can enhance the balanced performance of further R&D projects on renewable energy. It is expected to provide policymakers with insights to recognize the stages of innovation and commercialization and make resource allocation more efficient.

Further studies need to consider various nations’ R&D projects. Many countries also try to achieve appropriate R&D performance within the renewable energy area. Experts may want to know among nations’ relative performances and each nations’ R&D features. Combining parametric methods is also needed. Such as propensity score matching(PSM) and synthetic control methods are applied with DEA to overcome DEA’s weakness, such as treating external factors. Lastly, this paper was used until 2014 finished R&D projects. If the future study can use the updated data, the main findings may provide meaningful implications to policy decision-makers and scholars.