The innovative dimension of the research training programmes under H2020-MSCA-ITN1: a methodological approach to track, measure and analyse innovative aspects and provide policy-feedback conclusions.

Methods outline

We identified the appropriate population (sample) of H2020-MSCA-ITN projects to be analysed for innovation outputs, and then the nature and type of data to be collected for the analysis as well as the best way to collect the necessary data. Based on the literature analysis, we proposed and defined the appropriate indicators (KPIs and metrics) in order to track, analyse and present in a quantifiable and qualitative manner innovative elements in the H2020-MSCA-ITN projects.

A first analysis was done on the full H2020-MSCA-ITN portfolio covering the calls from 2014-2020. The source that we used to make the first extraction of data was the EC CORDA database (SAP BusinessObjects Business Intelligence, universe: EGRANTS context: H2020). Since the purpose of the study was to track innovation and similar outputs that have been already generated by the projects, it was decided to narrow the analysis on the population of projects under the calls 2014-2017. Projects under H2020-MSCA-ITN have a normal duration of 4 years (48 months) and with the selected sample of projects from the calls 2014-2017, the purpose was to analyse generated results from the already closed or in the closing mode projects (the analysis started in October 2021). In the closed projects, the interim and final reporting had already been conducted (contractual requirement), which ensured that the generated innovations and similar results were encoded in the EC corporate systems (databases) in order to allow their retrieval and further analysis.

We proceeded to the analysis of the H2020-MSCA-ITN calls 2014-2017 projects, again using the CORDA database/tool. From the extracted population of projects, a further analysis was carried out in order to identify those projects that appeared to have generated or declared innovation. By ‘generated innovation’ we refer to innovation and patents declared by the projects (and encoded in a structured manner in the CORDA database, following the project closure).

Finally, for the calls 2014-2017 projects, an analysis using the defined metrics for tracking and measuring generated innovation, has been performed on a set of data and documents comprising the Horizon Dashboard⁶ reports, the available H2020-MSCA-ITN project reports and relevant documents (i.e. progress, periodic and final project reports, Expert or Project Officer Assessment reports, ESRs’ questionnaires⁷ …) for tracking non-tangible innovation, and a set of EC corporate tools and services like databases (CORDA, CORDIS⁸), the Qlik Sense⁹ graphical data representation tool, the IR platform and the HRB service.

To summarize, the analysis of the extracted sample of H2020-MSCA-ITN projects was performed using:

Steps/components of the methodology

1. Reported patents and innovation outputs (CORDA and Qlik Sense/HorizonDashboard)

We used the CORDA database and the Qlik Sense platform tool for the data extraction. The extracted data were in the form of Microsoft Excel^@ spreadsheets.

H2020-MSCA-ITN projects have been generated via 7 calls (1 call per year) that have been performed under the seven years of the H2020 MFF (calls 2014-2020). The H2020-MSCA-ITN portfolio comprises 1034 executed (on-going or closed) projects. A focus on the 2014-2017 calls was considered as more relevant in order to ensure generated innovation from the closed or in the closing mode projects.

Based on the 2014-2017 projects portfolio, a data extraction was done from CORDA database in order to retrieve those projects that appear to have generated some kind of innovation or patent. The extraction of this information was done based on declared innovation and patents, using similar indicators that were provided by the data fields and filters of the CORDA database interface. In the data extraction, we included information fields related to - among others - the project number, call-id, mode, acronym, status, number and description of innovations, number and description of patents and number of publications (see Figure 1.a.1).

Figure 1.a.1. Example of data fields extracted from the CORDA database for 2014-2017 projects.

From the extracted data, a filtering of the projects was done based on declared innovation(s) and/or patent(s), resulting in 118 retained projects (MSCA ITN 2014-2017 innovation and patents [Data set], 2023a).

2. The Innovation Radar platform and innovation capacity indicators.

In order to further validate that the selection of 118 projects is the best and promising candidate for analysing and measuring innovation in the H2020-MSCA-ITN projects (beyond the patents), we used the Qlik Sense platform to extract additional Innovation Radar (IR) information for those projects.

We have further utilised the medium/high innovator definition of innovation radar and applied to the MSCA selected projects. The goal of the Innovation Radar Initiative is to identify high-potential innovations and innovators in EC-funded research projects, seeking to direct project consortia on the proper steps to reach the market. Its goal is to maximise the results of public funds mobilised on research. Two indicators based on the Innovation Radar data (Innovation Radar, 2018) have been defined: the Innovation Potential Indicator (Figure 2.1), which measures the projects’ innovation development towards commercialisation within the framework programme, and the Innovator Capacity Indicator (Figure 2.2), which captures the innovative capacity of the innovators behind these innovations. The Innovation Potential Indicator incorporates three essential indicators in the innovation development process: the Innovation readiness (related to the technical maturity of the innovations e.g. prototyping, demonstration or testing activities or a feasibility study), the Innovation management (related to the project consortium capability and its commitment to bring innovations to the market, e.g. preparing a business plan or market study, defining the IPR ownership, securing capital investments, engaging end user), the Market potential (related to the demand and supply side of innovations e.g. evaluating the market conditions for successful commercialisation, assessing how products or services satisfy a market sector and a potential customer base or potential barriers from the supply side which could jeopardise the commercial exploitation of innovations). The Innovator Capacity Indicator includes two indicators that capture the capacity of innovators in delivering successful innovations: the Innovator’s ability (related to the ability of organisations in developing innovations within the EC-funded activities e.g. the number of times organisations have been identified as key innovators by the Innovation Radar, the reviewers’ opinions about innovators’ potential and independence in fulfilling the market potential of innovations), the Innovator’s environment (related to the performance of the project in terms of innovations, the overall conditions in the project consortium which an innovator faces e.g. the commitment of relevant partners to exploiting innovations, the presence of organisations that are directly interested in exploiting the innovations, the positive ecosystem to facilitate the knowledge spill-overs between innovators and their environment).

Figure 2.1. Innovator Capacity Indicator.

Figure 2.2. Innovator Potential Indicator.

Categories of Innovators

The Innovation Radar (Innovation Radar, 2015) introduces three categories of innovators: Low, Medium and High Potential innovators. The assignment to a category is based on mean and standard deviation (SD) values of the Innovator Capacity Indicator (ICI) for innovators and uses percentile ranks. Ordering innovators into three different categories based on percentile ranks allows their performance to be compared very clearly with the remaining innovations and innovators in the sample. The percentile rank of an innovation or an innovator is defined as the percentage of innovators in the same sample that obtained a score at the same level or below that of the innovator’s score. In formal terms, the assignment of inventors to three categories is based on the following rules:

We have extracted from the Innovation Radar database the ICI assessment conducted on all MSCA ITN projects selected for funding from the calls 2014-2015-2016 and 2017, and compared with all project ICI assessment of the rest of H2020 calls from 2014 to 2017.

3. The HRB Service 4¹¹ (Module A: publishable reports of R&I project clusters analysis) as a proof of concept of the methodology by external experts.

We made use of the HRB Service 4 for the cross-validation of the - as they appear at the initial extraction - promising innovative results that were identified in the selected sample of projects. The main objectives of the HRB analysis were to:

The HRB analysis was based on the extracted (from CORDA) portfolio of the 118 selected projects.

The desk analysis and processing of the collected data was assigned to six independent and competent experts contracted by the HRB service. These experts were provided with the necessary data of the 118 projects (innovations and patents), the IR reports/questionnaires which had been already extracted by the IR platform and with additional project-related documents (like reports, deliverables, description of work, etc.) which have been extracted from the EC corporate databases (resulting in thousands of attachments to be considered and analysed for the total of 118 projects). Samples of these input data can be found under “Dataset, metrics and indicators used for Horizon Results Booster analysis” (Bitsios et al., 2023b).

The HRB analysis first focused on the mapping of projects, based on country of operation, thematic priorities and objectives. This allowed the clustering of projects into a few scientific domains. The scientific domains were determined via an initial analysis of the fields of science provided in the CORDIS database. When this information was not sufficient to assign a given project to one specific scientific domain, further analysis was done on the project’s definition and outputs, to assign every project into one identified domain.

The second step was to map the nature of the organisations and entities that were engaged in the 118 projects. This mapping was based on criteria like country, activity type, number of participations, consortium size, ‘Innovators’ (based on the IR definition) as project participants and per scientific domain and LERU participation¹². This analysis provided key insights on the type and participation of the organisations in the 118 projects and led to very interesting conclusions on the mix of the so-called ‘Innovators’ among the sampled projects participants.

Finally, the definition of a set of indicators/criteria permitted the extensive analysis of quantitative aspects of the clustered projects’ best practices and innovative outputs, as well as their qualitative assessment (HRB report, 2023).

The following sources were considered for the definition of the criteria and indicators for the processing of the clustered projects’ data:

As already mentioned under the ‘Limitations’, interviewing of the project stakeholders was not required in the end. The variety and amount of data that were collected by the rest of the sources/metrics were more than sufficient for the HRB analysis.

A summary of the set of criteria and indicators that were defined for the quantitative and qualitative assessment of the clustered projects’ best practices and innovative outputs, can be found under “Dataset, metrics and indicators used for Horizon Results Booster analysis” (Bitsios et al., 2023b).

4. The analysis of most frequently found terms (word clouds) which can be retrieved from the generated publications.

The dataset consisted of the MSCA ITN projects from the call years 2014 to 2017. Similarly, the reported scientific publications for the same projects have been retrieved. We carried out a wordcount on the titles/keywords of the projects, as well as on the titles of scientific publications. The analysis was performed by scientific panels (Chemistry – CHE; Economics – ECO; Engineering and Information Sciences – ENG; Environment and Geosciences – ENV; Life Sciences – LIF; Mathematics – MAT; Physics – PHY; Social Sciences and Humanities – SOC), as the MSCA programme is bottom-up by nature and funds research in all fields of research. Due to their reduced sizes, the ECO and SOC panels were merged in a single panel (ECOSOC) and the same was done with the MAT and PHY panel (MATPHY). The analysis was carried out in R (R Core Team), with word sizes being proportional to their frequency and displayed in word clouds. Some correlation analysis was also performed between the results obtained among funded proposals and those among publications in order to verify if the trends at funded proposal stage (beginning of the project) can be a good predictor of the work done and reflected in the corresponding publications. The 50 most frequent and correlated words are displayed in each figure. Most of the generic terms, considered as noise, were removed from the analysis. The same analysis has also been performed on the subset of the 118 projects flagged for innovation.

5. The non-tangible innovation and best practices results which can be retrieved from the non-structured data in the different project reports and documents.

Several indicators and best practices were selected to analyse non-tangible innovation elements in the 2014-2017 ITN projects, such as social innovation, innovative training practices, innovative project management, long-lasting collaborations etc. The type of data that was missing and could not be retrieved automatically from our various databases was further defined. Most of the best practices are also linked to data that can only be collected manually from the projects’ reports and documents. In order to ease the checks and screening of projects, some categories of data were pre-defined for each selected indicator and best practice.

6. Questionnaires/surveys from the ESRs that participated in those projects.

As an indicator of non-tangible innovative results, the MSCA follow-up questionnaires after the fellowship and two years after the end of the fellowship were examined. Respondents were asked for feedback regarding their training experience and career improvement. A total of 4431 respondents answered to the end of the fellowship questionnaire, while a total of 181 fellows responded to the questionnaire 2 years after the end of the fellowship. The questionnaires were anonymous; therefore, none of the replies can be traced back to a particular individual. The results presented in this document reflect the answers retrieved from these 4431 and 181 survey replies.

Limitations

Past research on innovation (OECD, European Union, 2018; Taques, López, Basso, Areal, 2020), already proposed a number of indicators for measuring innovation in different application domains. It must be noted that, for every identified indicator there are advantages and disadvantages on its use for measuring innovation. For example, patent databases can be useful when assessing innovation impact for research projects. On the other hand, while the use of questionnaires to track and measure innovative outputs allows identification of the type of innovation generated by a given organisation, it has certain drawbacks. It is prone to subjectivity (there is no control point for the information because the input is given by the same organisation claiming the innovation generation), not all innovation elements are reported due to information confidentiality concerns, and it is also prone to errors (typos, clerical errors) during the completing process (Taques, López, Basso, Areal, 2020).

In the present study, another limitation concerns the collection of unstructured data, manually retrieved from various project reports and documents. A methodology and pre-defined categories were discussed in order to limit as much as possible the subjectivity and the variability in the data collection exercise.

Regarding the population/sample of the projects that were analysed, we focused only on closed projects (from calls 2014-2017). For closed projects, normally the interim and final reporting has been conducted and the generated innovations and similar results have been encoded in the corporate systems, thus can be further analysed. Consequently, innovations and similar early results from on-going projects were not considered for this analysis.

The selection of the two main indicators (patents and innovations) for the identification of the sample of projects for the analysis may limit the representation of Social Sciences and Humanities (SSH) – related projects in the final selected sample. This can be considered as expected for two reasons: The generation of patents and similar innovation results are more likely to happen from innovators that are industrial and/or commercial in their nature (HRB report, 2023). Similarly, the use and submission of patents and similar innovations, is traditionally reserved to domains of science and technology, like the STEM (Science, Technology, Engineering, Mathematics) and medicine/biomedicine/drugs disciplines (HRB report, 2023; Reymond, 2020). However, relevant research has shown for example, that an in-depth analysis in patent sources can provide useful information on SSH topics beyond those of mere technological intelligence (Reyomond, 2020).

As far as the HRB analysis is concerned, among the initially defined sources and metrics, interviews with selected project stakeholders were foreseen. However, interviewing project stakeholders was considered time-consuming while at the same time, the abundance of data collected by the rest of the defined sources made clear that there was no real need for interviews to complement the collected data.

Certain limitations and assumptions apply to the information and outputs presented in this study since the CORDA database and Innovation Radar database are not wholly consistent nor harmonised across projects’ data. As such, the amount and granularity of information available may vary.

Analysis on patents and innovation outputs (CORDA and Qlik Sense)

As mentioned earlier, a focus on the 2014-2017 calls was considered as more relevant. The H2020-MSCA-ITN 2014-2017 portfolio comprises 550 executed (on-going or closed) projects (see Figure 1.b.1).

Figure 1.b.1. Number of H2020-MSCA-ITN projects under 2014-2017 calls (the 552 population includes 2 terminated/rejected projects).

From those projects, the vast majority (77%) were ETNs, while the rest were EIDs (16%) and EJDs (7%) (see Figure 1.b.2).

Figure 1.b.2. Mode distribution (ETN-EID-EJD) under 2014-2017 calls.

The distribution of modes in the 2014-2017 projects portfolio is in coherence with the distribution of modes in the total H2020-MSCA-ITN projects population from all calls 2014-2020 (78% were ETNs, 15% were EIDs and 7% were EJDs). Moreover, an analysis of the 2014-2017 population of projects based on the H2020 KPIs, like the number of publications and patents per 10 million euros funding, revealed that, in terms of publications, MSCA-ITN exceeded the established target. As for patents, although this indicator is not applicable to MSCA under the Excellence pillar of H2020, H2020-MSCA-ITN 2014-2017 projects showed a good performance in terms of the ratio of generated patents. (see Figure 1.b.3).

Figure 1.b.3. Analysis of performance of H2020-MSCA-ITN 2014-2017 projects based on the H2020 KPIs (generated publications and patents per 10 million euros funding).

From the extracted data, a filtering of the projects was done based on declared innovation(s) and/or patent(s). Out of the 550 analysed projects, 432 declared no generated innovation or patent, while the remaining 118 projects declared the generation of at least one innovation output or patent or both. Since the focus of this study was to measure the innovative dimension under H2020-MSCA-ITN, a deeper analysis was performed on the 118 projects with only innovation or patent output or with both. The analysis showed that out of those 118 projects which declared the generation of at least one innovation or patent output, 93 projects declared at least one generated innovation and no patents, 17 projects declared at least one patent and no innovation, and 8 projects declared at least one innovation and one patent (see Figure 1.b.4). It must be noted that ETN mode contributes to all types of innovation outputs (only inno, only patent, inno and patent).

Figure 1.b.4. H2020-MSCA-ITN projects under 2014-2017 calls that generated at least one innovation output and/or at least one patent and additional breakdown of these projects (118) based on the occurrence of only innovation (93) or only patent (17) or both (8).

We further examined the mode distribution in the 118 projects with the sole purpose of verifying that they can be considered as a good representative sample of the total 2014-2017 population of projects. Indeed, out of these 118 projects, 92 projects (~78%) were ETN, 20 projects (~17%) were EID and the rest, 6 projects (~5%), were EJD (see Figure 1.b.5). This mode distribution among the 118 selected projects is similar and comparable to the mode distribution among the total number of 2014-2017 calls’ projects (see Figure 1.b.2).

Figure 1.b.5. Mode distribution (ETN-EID-EJD) in the 118 projects with innovative dimension.

In terms of patent performance based on the project mode, the analysis of 118 projects showed that EIDs included in this sample, tend to report proportionally more patents per EID project (~1 patent/EID project) than ETNs of the same sample (see Figure 1.b.6). This can be an encouraging early finding on the effect of the project mode (EID, more industry-oriented) to the generation of patents.

Figure 1.b.6. Number of patents per flagged project for innovation (118 projects) (normalized project relative size).

Based on the findings of this first component of the methodology, we are already in the position to consider the selected 118 projects as a very good and representative sample to measure, analyse and draw conclusions on the innovative aspects of the H2020-MSCA-ITN projects. As already mentioned, additional components were considered in the methodology to further validate the credibility of the selected sample, and these are elaborated in the following paragraphs.

The Innovation Radar indicators

a. Innovation Radar analysis

In order to further confirm that the selection of the 118 projects is the best and the most promising candidate for analysing and measuring innovation in the H2020-MSCA-ITN projects (beyond the patents), we extracted additional Innovation Radar (IR) information for those projects (see Figure 2.b.1).

Figure 2.b.1. Data on generated innovation for the 101 projects (out of the selected 118 projects).

It is to be noted here that the extracted information from Qlik Sense platform on generated innovation for the selected 118 projects, is actually restricted to 101 projects for the H2020-MSCA-ITN 2014-2017 calls. This is logical, since 17 projects out of the 118 selected projects (see Figure 1.b.4) have reported only the generation of patents and no innovation outputs (at least in the sense of innovation outputs that are captured via the IR platform).

b. Innovator Capacity assessment, comparison between MSCA and the rest of H2020

We have extracted from the two Innovation Radar Platform databases (up to April 2018 and from May 2018 onwards) the ICI assessments conducted on those 101 projects coming from MSCA ITN calls 2014-2015-2016-2017 and all other H2020 projects from the same period (calls 2014 to 2017).

We calculated, according to the Innovation radar methodology, the average ICI and standard deviation of the H2020 selected population and MSCA one. These values computed as Average +/- Standard Deviation, determine the ICI values that define the low, medium, high innovator categories in both population under analysis. These values and the percentage of the organizations in each category are reported and compared in Table 2.b.1.

From the comparison, it is clear that the two populations performed equally in terms of innovators’ distribution per innovation category, showing similar percentage of medium to high innovators vs the total innovators. We have to underline that MSCA being under the excellence science pillar, ITN work programmes involve young researchers with less than four years research experience employed in training networks mainly focused on providing the necessary knowledge and expertise to boost the young researchers’ career prospects within an innovation ecosystem, so the fact that MSCA ITN shows the same innovations character of H2020 programmes whose innovation outcome are coming from Research and Innovation Actions (RIA) and Innovation Actions (IA) projects where the main players are senior researchers, scientists employed in academic and non-academic organisations, is a truly remarkable achievement of the ITN actions. However, it could be expected if considering the following analysis of the common innovators in both populations as follow-up of the HRB study.

c. Follow-up of HRB study on H2020 and MSCA ITN common innovator organisation type analysis

Following the results of the HRB analysis, conducted on the 118 projects, we have developed on the ‘innovator’ concept, based on the Innovation Radar definition for the 101 projects assessed by Innovation Radar out of the 118. According to the HRB study:

We have further investigated this important result, analysing the organisation type of those innovators present in 101 out of the 118 projects studied by HRB, in comparison with the entire population of innovators in H2020 projects originated from the same calls of reference. Out of 265 organisations considered as innovators in MSCA ITN, 240 are also innovators in H2020. We have used the ICI values of H2020 to define the innovator categories (low, medium and high) of those common innovators per organisation type; academic (HES, RES, PUB)¹³ and non-academic (PRC) differentiating within the non-academic sector group between SME and non-SME. The distribution of organisation type is shown in the following Table 2.b.2.

The pie chart analysis per innovator category (see Figures 2.b.1a, 1b, 1c) shows that the higher the innovators ICI of one organisation, the higher the concentration of SME within a certain innovator category, confirming once more the finding of the HRB study. In the high innovator band, we found 28% of SME while in the medium one 18%, and in the lower one, only 9%. This is clearly visible in the pie charts for each innovators category. This is somehow expected since the high innovation character is linked to high technology readiness level, which is an essential element pursued by organisations who want to be closer to the market valorisation and thus adapt their exploitation plan according to the projects they are in. Non-academic organisations and particularly SMEs are indeed more interested in high innovation that brings them closer to market valorisation. In fact, the lower the ICI and the higher is the percentage of non-profit organisation, namely universities, research and public bodies.

Figure 2.b.1a. High Innovator in MSCA-ITN per organisation type.

Figure 2.b.1b. Medium Innovator in MSCA-ITN per organisation type.

Figure 2.b.1c. Low Innovator in MSCA-ITN per organisation type.

To highlight this finding, out of the 13 organisations defined as high innovators in both MSCA-ITN and H2020, 9 (69%) are SMEs as shown in Figure 2.b.2. This once more confirmed the interest of the non-academic sector (mainly SME) in being at forefront of the innovation.

Figure 2.b.2. Organisation type of high Innovators in MSCA-ITN which are also high innovators in H2020.

Horizon Results Booster (HRB) analysis (proof of concept)

The HRB analysis (HRB report, 2023) generated a number of very interesting results for the portfolio of 118 projects funded by the H2020-MSCA-ITN Programme taking into account four consecutives calls 2014-2015-2016-2017. This section summarises the key results and findings.

The projects and their outputs were clustered into 15 scientific domains. The clustering of the projects into the scientific domains allowed more efficient identification of commonalities amongst the projects and the comparison of the innovative elements between these scientific domains. These domains and the distribution of projects (in total 118 projects) can be seen in Figure 3.b.1.

Figure 3.b.1. Scientific domains and distribution of clustered projects (for the 118 projects).

The mapping exercise of the organisations that were engaged in the 118 projects revealed among others very encouraging results in terms of the innovation potential of the involved organisations and the frequency of their participation in the H2020-MSCA-ITN action. The results indicated that 628 organisations participated in the analysed projects and while most of the organisations (almost 73%) were involved in only one analysed project, more than one-fourth (~27%) of the organisations demonstrated more active engagement. In terms of countries’ representation among the organisations involved in the analysed projects, 475 (out of the 628 in total, almost 76%) are located in the EU-14 countries¹⁴, 122 in non-EU (~19%), and only 31 (almost 5%) in the EU-13 countries¹⁵. From the engaged organisations, 60% of the project participants represent higher education institutions and research organisations (276 and 98), while 238 (38%) are companies from the private sector. The analysis showed that a project consortium is typically built of eight organisations and on average it covers four higher education institutions, three private enterprises, and one research organisation. Of the 171 organisations that joined more than one project, 100 fall under the category of ‘Innovators’ and most of the projects had at least one innovator in their consortium. From the total number of ‘Innovators’ identified in the 118 projects, 77% are located in the EU-14 countries, 3% in the EU-13 countries and 20% in non-EU countries. Similarly, as for the project participants in general, 60% of the ‘Innovators’ represent higher education institutions and research organisations and 39% are companies from the private sector. On average, four ‘Innovators’ participated in one project and in most projects, at least one ‘Innovator’ was involved. Moreover, and regarding ‘Innovators’ amongst the defined (15) scientific domains, they are well represented as participating organisations across all scientific domains. Interestingly, the mapping exercise of the involved organisations showed that the members of the League of European Research Universities (LERU) network are sufficiently represented in most scientific domains (see Figures 3.b.2, b.3, b.4, b.5, b.6, b.7).

Figure 3.b.2. Participants’ engagement in the analysed projects.

Figure 3.b.3. Participating organisations per type of activity and country.

Figure 3.b.4. ‘Innovators’ per type of activity and country.

Figure 3.b.5. The representation of ‘Innovators’ in the projects. For example, in 27 projects (out of the 118 projects) 4 ‘Innovators’ were part of the project participants. Only in 2 projects (out of the 118) no ‘Innovator’ was part of the project participants.

Figure 3.b.6. Share of innovators of all participants per scientific domain.

Figure 3.b.7. Share of LERU members of all participants per scientific domain.

Concerning the results for innovative outputs (i.e. patents, publications, innovations from the IR service), the analysis revealed that certain scientific domains are more prone to generating patentable outputs, possibly due to their results having competitive market applications. In terms of publications, all scientific domains have significant output in terms of peer-reviewed articles, some are more prolific in this regard than others. This could suggest possible differences between the domains regarding certain factors, for example the novelty of the research area, the availability or feasibility of research and experimentation, or the internal dynamics of domain research communities. Finally, the innovation output performance as an average number of innovations was identified and was presented, per scientific domain, which can suggest the innovation potential of a domain based on current research challenges etc. (see Figures 3.b.8, b.9, b.10).

Figure 3.b.8. Average number of patents per scientific domain.

Figure 3.b.9. Average number of peer-reviewed articles per scientific domain.

Figure 3.b.10. Average number of innovations identified by the Innovation Radar per scientific domain.

Finally, and in terms of the capability of the analysed projects and/or their results, to be transferred elsewhere, to be applied elsewhere or to be able to engage in synergies with other projects (so to be considered as innovative/best practices), the HRB analysis provided remarkably interesting results. For example, projects with large numbers of events, particularly conferences, were found in almost every scientific domain, which can suggest the effort for public outreach of the projects (integral part of the H2020-MSCA-ITN programme) in order to ‘transfer new knowledge to scholarly communities and communicate it to society’, in line with good practice elements for doctoral training (LERU advice paper, 2014). Regarding performed secondments (also an integral part of the H2020-MSCA-ITN action), the qualitative feedback provided by the participating projects in the Innovation Radar questionnaires (as part of the analysed documents), indicated that the projects consider secondments to be important and would like them to be further supported and enabled. These elements can be assumed as part of the non-tangible innovation that can be observed in the H2020-MSCA-ITN projects, where best practices, such as the mobility of the recruited researchers (for example via the secondments), the participation in wide training events, or the cross-sectoral cooperation (via the exchange of researchers in academic and non-academic participants) nurture the co-creation of novel solutions (which are then reported as innovation outputs). The diffusion and transfer of knowledge, and the cross-sectoral cooperation, have been put in place to stimulate innovation creation.

The detailed results and interesting findings of the full HRB analysis can be found in the published HRB report (HRB report, 2023), which was generated in the context of this study. The HRB analysis led to key conclusions and concrete links to policy-feedback suggestions and recommendations, which are presented in the ‘Conclusions and main recommendations’ section.

Analysis of most frequently found terms (words clouds) which can be retrieved from the generated publications

The purpose of this analysis was to assess, using a visual representation, what the most frequent words and topics in MSCA ITN projects are and if some trends identified at the project level (intended research) can be a good predictor of the effective research performed (outputs such as scientific publications).

It must be noted here that, as it can be observed in Figure 1.b.3 (related to the extraction of the calls 2014-2017 projects), the performance of the H2020-MSCA-ITN programme in terms of publications output was above the average in comparison to the whole H2020 programme. The beneficiaries of those 550 projects from the H2020-MSCA-ITN 2014-2017 calls, have reported (as of December 2021) a total of 8883 scientific publications (Table 4.1).

ENG and LIF panels represent almost 60% of the total (Table 4.2). When looking at the scientific productivity of ITN projects, the two largest panels (ENG and LIF) have the highest shares of publications in absolute terms, but the Chemistry (CHE) and Mathematics/Physics (MATPHY) panels are more productive in relative terms (with an average of 24,3 publications/project for MATPHY).

In the following figure (Figure 4.1) we represent the correlated words frequencies observed both in projects’ titles and keywords and in the respective publications titles, per panel.

Figure 4.1. Correlated words frequencies in projects’ titles and keywords, and in the respective publications titles, per panel: A: CHE panel; B: ECO-SOC panels; C: ENG panel; D: ENV panel; E: LIF panel; F: MAT-PHY panels.

It should be noticed that in most of the scientific panels, some specific words appear more frequently, giving some scientific trends both at project and publication stages. For example, the most frequent words from the projects and linked publications in the Chemistry panel are mainly related to this scientific domain, such as synthesis, materials, drug, organic and molecular chemistry, while projects and linked publications from the Engineering panel focus more on design, model, systems, control and energy aspects. Similarly, the Environment panel shows some good correlation between the projects and linked publications mainly in environment-related keywords, such as modelling, plant, food, water, processes, energy, sustainable and industry. Interestingly, the Economics and Social Sciences panels show more interdisciplinarity with several frequent words related to health, development, care, innovation, policy and technology. Life Sciences panel shows a strong focus on cells, cancer, development, model, disease, drug, treatment and clinical aspects. Most of the frequent correlated words found in projects and linked publications from the Mathematics and Physics panels concern quantum and imaging-related keywords. Our analysis shows that in some specific scientific domains, such as quantum physics or cancer research, this tool seems to be a good predictor of what is effectively achieved and published by the projects.

In a second phase, the same analysis was performed on the subset of the 118 projects flagged for innovation (Figure 4.2). The results tend to show that the most innovative ITN projects and linked publications focus on models/modelling, cells, design, data, applications, systems, energy, imaging, health, surface and industrial aspects, suggesting a significant emphasis on applied sciences, as also suggested in the HRB portfolio analysis.

Figure 4.2. Correlated words frequencies in projects titles and keywords, and in the respective publications titles from the 118 projects flagged for innovation.

In this exercise, we obtained, using a text mining analysis, a visual representation of potential thematic domains where innovative outputs can be expected, as confirmed by other analyses and indicators defined in our methodology.

Non-tangible innovation and best practices results

Increasingly, more publications (Rupietta et al., 2018; Bader, Alharbi et al., 2019) raise the importance of measuring and monitoring not only tangible innovation but also more organisational innovation elements, which can also have a crucial impact on social, environmental or policy aspects, as well as on the career development, knowledge transfer and cooperation aspects in an organisation. All these aspects are also considered as “signature” principles for MSCA. Therefore, it was important to analyse these less tangible innovation elements in the ITN projects and investigate if some correlations could be identified with the projects reporting more tangible innovation.

The selected indicators and best practices are ranging from social innovation, innovative training practices, involvement of SME/industry in the projects to innovative supervision, networking opportunities, long-lasting collaborations, synergies and policy-driven results.

First analyses show that for social innovation (Figure 5.1), 22% of all 550 ITN projects report health-related measures, followed by green-related measures (9,3%). Mobility and transport policy, family-friendly conditions or special needs/disability measures are less reported by the projects, compared to health or green-related measures. This could be explained by the recent covid-19 pandemic which impacted most of the H2020 ITN projects, as well as by the Green Deal launched by the European Commission, encouraging the greening of all projects.

Figure 5.1. Social innovation aspects reported in H2020-MSCA 2014-2017 ITN projects.

Innovative training practices are also a key element of MSCA ITN projects. Interestingly, the Figure 5.2 shows that 28,4% of the projects report transversal training with a long-lasting impact that stimulates innovative thinking and enhances the researchers’ career, through for example entrepreneurial, creative thinking or teamwork aspects. An additional important element is the structural effect and impact of long-lasting training on the participating organisations, reported by 18.1% of the projects. 16.1% of the projects reported some innovation in terms of the training audience and a wider impact, while 11.5% reported innovative open tool for enhanced training, such as e-learning platforms.

Figure 5.2. Innovative Training Practices reported in H2020-MSCA 2014-2017 ITN projects.

As reported earlier in the study, the participation of identified innovators plays a key role in the innovative potential of an ITN project, a significant share of them being from the non-academic sector (in particular SME/industry). It was therefore interesting to analyse the role and active contribution of SME and industry participants in H2020 ITN projects. The Figure 5.3 confirms that a large number of projects report a strong contribution of their SME and industry participants, in particular in the secondments of researchers, the training offered to the researchers and their supervision. Their involvement in the exploitation of results is also reported by almost 20% of the projects.

Figure 5.3. SME/industry active contribution reported in H2020-MSCA 2014-2017 ITN projects.

Non-tangible innovation in supervision seems to be less reported by the projects (Figure 5.4). It should be noted, however, that some projects showed a strong engagement in improving those crucial aspects in the researchers’ training and described very good practices, such as the pairing of experienced and new supervisors, dedicated training of supervisors, a new feedback system from fellow to supervisor, or an internal network or platform of supervisors to exchange knowledge and best practices. Better results could be expected at the end of H2020 and under Horizon Europe with the recently published MSCA guidelines on supervision¹⁶.

Figure 5.4. Innovative supervision aspects reported in H2020-MSCA 2014-2017 ITN projects.

Not surprisingly, almost half of the projects report scientific networking opportunities for the ESRs, followed by cross-sectoral networking opportunities and professional networking opportunities, which is one of the objectives of the MSCA ITN programme (see Figure 5.5).

Figure 5.5. Networking opportunities reported in H2020-MSCA 2014-2017 ITN projects.

New and enhanced research results often come from synergies and synergistic effects. Synergies were also analysed in H2020 ITN projects, as another non-tangible innovative indicator. The Figure 5.6 shows that around 15% of the projects report scientific collaboration within the same domain with other ITN projects, while almost 10% of the projects report the organisation of common training or dissemination activities with other ITN projects. Interdisciplinary scientific collaborations (e.g. between ITN projects in different scientific panels) account for almost 8% of the analysed projects.

Figure 5.6. Synergy aspects reported in H2020-MSCA 2014-2017 ITN projects.

Almost 30% of the ITN projects report co-authoring articles amongst the supervisors and their ESRs. 23,2% of the projects also report co-authoring articles amongst ESRs, which is a positive result since joint publications are very much encouraged in all ITN projects. It often demonstrates the relevance of the joint research project and how well the individual research projects, performed by the ESRs, are interconnected. In addition, the Figure 5.7 exhibits that almost 7% of the project report the creation of spin-off companies, while around 5% create strong collaborations with third countries.

Figure 5.7. Long-lasting collaborations reported in H2020-MSCA 2014-2017 ITN projects.

In the context of more and more requests of feedback to policy, another indicator linked to policy-driven results was analysed in ITN projects (see Figure 5.8). A significant share of the projects report policy-driven research, such as generating data for making policy decisions (18,3%), or organising high-impact conferences, clustering events, etc … (13,5%). For 10,9% of the project, the high policy impact is at the level of researcher who produces results that can feed into policy making etc …

Figure 5.8. Policy driven results reported in H2020-MSCA 2014-2017 ITN projects.

Interestingly, if we compare our sample of 118 ITN projects flagged for tangible innovation with the other ITN projects, the Figure 5.9 below shows that the projects already reporting tangible innovation tend to also report more non-tangible innovation and best practices. The results were particularly significant for social innovation, innovative training practices or SME/Industry contribution, networking opportunities for the ESRs and synergies aspects.

Figure 5.9. Comparison of non-tangible innovation elements and best practices between the total H2020-MSCA 2014-2017 ITN projects and the 118 projects flagged for tangible innovation.

The percentages can be above 100% since projects can report more than one innovation/best practice per indicator.

Fellows’ questionnaires

As another source of non-tangible innovation, MSCA follow-up questionnaires after the fellowship and two years after the end of the fellowship were examined.

After their fellowship, Marie Skłodowska-Curie Actions (MSCA) fellows are invited to complete two short surveys covering several aspects during and after the fellowship, including the impact of the fellowship on their skills development, career and employability. The evaluation questionnaire is completed soon after the end of their fellowship to assess their experience, skills developed and immediate next steps after their MSCA project. The follow-up questionnaire is submitted two years after the fellowship to gather further information on the more mid and long-term impact of the fellowship and the career paths of the fellows.

Regarding the questionnaires at the end of the fellowship, we also compared the respondents’ total answers (4431) with the ones from the 118 ITN projects (1236) who declared the generation of at least one innovation or patent. The comparison of our sample of 118 ITN projects flagged for tangible innovation with the other ITN projects showed very similar results, as demonstrated below.

Recent publications highlighted several important indicators of non-tangible innovation, such as the career development and job satisfaction level of the researchers. These aspects could be analysed from the various questions included in the questionnaires. In particular, 87% of the respondents declared that their fellowship within the ITN has had a very positive impact on their professional development (very good or good impact). The trend of replies for the 118 ITN projects flagged for innovation was very similar to the general one (as shown in Figure 6.1 A&B).

Figure 6.1. Impact on professional development.

A: total replies; B: 118 projects flagged for innovation.

When asking if the job was research/innovation-related, the majority of the respondents (87%), both from the overall projects sample and the 118 projects flagged for innovation, worked in a research/innovation research job after their fellowship (Figure 6.2).

Figure 6.2. Job research/innovation-related.

A: total replies (Yes: 92%; No: 8%); B: 118 projects flagged for innovation (Yes: 93%; No: 7%).

Regarding the specific impact of the ITN fellowships on the researchers’ career, and when asking the fellows if they had found an employment at the end of the fellowship (within the first three months), a high share of fellows (around 42%) had an employment at the end of the fellowship in both samples (total ITN portfolio and 118 projects). In general, the fellows are still finalising their PhD three months after the end of their fellowship, therefore it can explain why almost 50% declared that they did not find an employment yet after their fellowship (see Figure 6.3). One remarkable finding is that among the 118 projects, 6 very best projects have patent and innovation, and the fellow employability is at 100 % (data not shown).

Figure 6.3. Employment at the end of the fellowship.

A: total replies (Yes: 42%; No: 48%; No Reply:11%); B: 118 projects flagged for innovation (Yes: 42%; No: 48%; No Reply: 10%).

Even more positive results could be observed when analysing the MSCA evaluation questionnaires two years after the end of the fellowship. In particular, out of the 181 respondents, more than 75% declared that they were employed or self-employed and 12% were still involved in education and training activities, as shown in Figure 6.4.

Figure 6.4. What is the current main activity of the ESRs.

The majority of the fellows considered that their MSCA fellowship played an important role in obtaining their current position. 55% of the respondents consider that their MSCA fellowship helped them to a large or very large extent (Figure 6.5).

Figure 6.5. Role of the MSCA fellowship in obtaining the ESR’s current position.

These results are very encouraging even for the follow-up questionnaires, two years after the end of the fellowship (181 replies), showing that most of the fellows are satisfied or very satisfied of their experience in an MSCA ITN project and that it significantly helped them improve their professional career, as illustrated by their overall satisfaction with their current professional situation (Figure 6.6).

Figure 6.6. Overall satisfaction of the ESRs with their current professional situation.

Very satisfied: 59 (33%); Satisfied: 91 (50%); Dissatisfied: 16 (9%); Very Dissatisfied: 4 (2%); No Reply: 11 (6%).

When asking if their work was research/innovation-related, the same number of employed respondents (134) confirmed that their job was research/innovation-related (Figure 6.7).

Figure 6.7. Is the ESRs’ job research/innovation-related 2 years after the end of the fellowship?

Yes: 74%; No: 11%; No Reply: 15%.

The use of questionnaires was meant to track and measure less tangible innovative outputs and notably concrete impact on the researchers’ career development through short and simple questions and appropriate indicators. It can be concluded that these types of questionnaires can be a good tool and benchmark for further improvement of MSCA work programme and policy feedback recommendations. They show how the innovative training network can enhance the employability of the fellows, especially in finding research and innovation-related jobs.

Conclusions

Regarding the validation of our methodology to track, measure and analyse innovative elements in EU-funded projects, we have performed an analysis of the metrics to be considered, based on the Horizon Dashboard reports, various types of H2020 MSCA ITN and various EC databases, focussing on the H2020 KPIs for high-impact publications, patents, allocated funding, innovation outputs, the Innovation Radar indicators, an analysis of most frequently found terms (words clouds), as well as non-tangible innovation and best practices results. The significance of our results validates the need of having a methodology in place (as proposed in this study) to track, measure and analyse innovative aspects and best practices. Measuring innovation is key for the creation and implementation of research and innovation-friendly policies that can further boost innovation creation within the EU, and our study revealed that the H2020-MSCA-ITN funding scheme has a clear and important contribution to this process.

A set of indicators and a methodology have been proposed in this study to allow the extraction of innovative aspects (tangible and non-tangible innovation) in EU-funded projects to support policy-feedback activities. These elements can serve as a reference and can be re-used with small adaptations to other EU-funded programmes to support similar activities. The exercise for the extraction and analysis of the 118 projects confirmed the validity of the defined indicators as well as of the components of the proposed methodology, since coherent results and findings were observed across the components of the methodology. These indicators can in turn act as predictors of potential success and innovation of ITN/DN projects or other EU-funded projects.

Recommendations

Beyond the excellence of the results and the high performance of the programme towards innovation generation, certain points for further improvement are identified (i.e. countries representation, scientific domains currently being less-represented in the innovation generation), which can feed EU-funding policy activities to improve innovation and excellence across Europe and beyond.