Achievements and Impact of the Big Data Value Public-Private Partnership: The Story so Far

Abstract

The European contractual Public-Private Partnership on Big Data Value (BDV PPP) has played a central role in the implementation of the revised Digital Single Market strategy, contributing to multiple pillars, including “Digitising European Industry”, “Digital Skills”, “Building the European Data Economy” and “Developing a European Data Infrastructure”. The BDV PPP and the Big Data Value Association have also played a pivotal role in the European Artificial Intelligence and Data Strategies launched by the European Commission in 2018. This chapter provides an overview and an in-depth analysis of the impact of the PPP by mid-2019, with a focus on the achievements and the overall impact since the launch of the PPP.

1 Introduction

The European contractual Public-Private Partnership on Big Data Value (BDV PPP) was signed on 13 October 2014. It marked the commitment of the European Commission, industry and partners from research to build a data-driven economy across Europe, mastering the generation of value from Big Data and creating a significant competitive advantage for European industry, thus boosting economic growth and employment. The BDV PPP started in 2015 and was operationalised with the launch of the Leadership in Enabling and Industrial Technologies (LEIT) work programme 2016/2017 of Horizon 2020 (H2020) with the first PPP projects (Call 1) starting in January 2017. With 57 projects,¹ an allocated investment of public funding of €301 million by the end of 2019 and around 300 organisations as part of the private association² (Big Data Value Association, BDVA) over the years, the Big Data Value PPP has played a central role in the implementation of the revised Digital Single Market (DSM) strategy, contributing to multiple pillars including “Digitising European Industry”, “Digital Skills”, “Building the European Data Economy” and “Developing a European Data Infrastructure”. The BDV PPP and the BDVA have also played an important role in the European AI and Data Strategies launched by the European Commission in 2018. This chapter provides an overview and an in-depth analysis of the impact of the PPP by mid-2019, with a focus on the achievements and the overall impact since the launch of the PPP.

This chapter details the achievements and the impact of the Big Data Value PPP. After explaining the key elements of the Big Data Value PPP in Sect. 2, and presenting a summary of the achievements and impact created during 2018 discussed in Chap. “A Roadmap to Drive Adoption of Data Ecosystems”, an in-depth analysis of the overall progress towards the mains goals of the partnership by mid-2019³ is given in Sect. 4. Finally, the Sect. 5 concludes with a summary and perspectives on the future.

2 The Big Data Value PPP

The vision, overall goals, main technical and non-technical priorities and a research and innovation roadmap for the European Public-Private Partnership (PPP) on Big Data Value are defined in the Big Data Value Strategic Research and Innovation Agenda (BDV SRIA) (Zillner et al. 2017).

The BDV PPP SRIA defined the roadmap and methodology by describing three different phases:

• Phase I: Establish an Innovation Ecosystem (H2020 WP 2016–2017 calls)

• Phase II: Disruptive Big Data Value (H2020 WP 2018–2019 calls)

• Phase III: Long-term Ecosystem Enablers (H2020 WP 2019–2020 calls)

The BDV SRIA has been regularly updated incorporating the multi-annual roadmap of the BDV PPP. BDV SRIA v4 (delivered at the end of 2017) provides direct input to the LEIT WP 2018–2020 as defined in its updated Phases II and III.

The BDV PPP projects cover Big Data technology, including Artificial Intelligence methods, and application research and innovation, new data-driven business models, data ecosystem support, data skills, regulatory and IPR requirements, and societal aspects. The value generated by Big Data technologies empowers Artificial Intelligence to foster linking, cross-cutting and vertical dimensions of value creation at the technical, business and societal level across many different sectors.

2.1 BDV PPP Vision and Objectives for European Big Data Value

The Big Data Value Association (BDVA) and the BDV PPP have pursued a common shared vision of positioning Europe as the world leader in the creation of big data value. The BDV PPP vision for Europe in 2020 has concerned the following aspects:

• Data: Zettabytes of useful public and private data will be widely and openly available. By 2020, smart applications such as smart grids, smart logistics, smart factories and smart cities will be widely deployed across the continent and beyond. Ubiquitous broadband access, mobile technology, social media, services and the IoT on billions of devices will have contributed to the explosion of generated data to a global total of 40 zettabytes. Much of this data will yield valuable information. Extracting this information and using it in intelligent ways will revolutionise decision-making in business, science and society, enhancing companies’ competitiveness and leading to the creation of new industries, jobs and services.

• Skills: Millions of jobs will become established for data engineers and scientists, and the Big Data discipline will be integrated into technical and business degrees. The European workforce is increasingly data savvy, regarding data as an asset.

• Legal: Privacy and security can be guaranteed along the big data value chain. Data sharing and data privacy can be fully managed by citizens in a trusted data ecosystem.

• Technology: Real-time integration and interoperability among different multilingual, sensorial and non-structured datasets will be accomplished, and content will be automatically managed and visualised in real-time. By 2020, European research and innovation efforts will have led to advanced technologies that make it significantly easier to use Big Data across sectors, borders and languages.

• Application: Applications using the BDV technologies can be built, which will allow anyone to create, use, exploit and benefit from Big Data. By 2020, thousands of specific applications and solutions will address data-in-motion and data-at-rest. There will be a highly secure and traceable environment supporting organisations and citizens, with the capacity to sustain various monetisation models.

• Business: One true EU single data market will be established, thus allowing EU companies to increase their competitiveness and become world leaders. By 2020 value creation from Big Data will have a disruptive influence on many sectors. From manufacturing to tourism, from healthcare to education, from energy to telecommunications services, from entertainment to mobility, big data value will be a key success factor in fuelling innovation, driving new business models, and supporting increased productivity and competitiveness.

• Societal: Societal challenges will be addressed through BDV systems, focusing on areas such as the high volume, mobility and variety of data.

The above-addressed aspects were planned to impact the European Union’s priority areas as follows:

• Economy: The competitiveness of European enterprises will be significantly higher compared to their worldwide competitors, due to improved products and services and greater efficiency based on the value of Big Data. One true EU single data market will be established, allowing EU companies to increase their competitiveness and become world leaders.

• Growth: A flourishing sector of expanding new small and large businesses will result in a significant number of new jobs focusing on creating value out of data.

• Society: Citizens will benefit from better and more economical services in a stable economy where data can be shared with confidence. Privacy and security will be guaranteed throughout the life cycle of BDV exploitation.

These three factors were designed to support the major EU pillars as stated in the Rome Declaration of March 2017 (European Council 2017): a safe and secure Europe; a prosperous and sustainable Europe; a social Europe; and a stronger Europe on the world stage.

2.2 Big Data Value Association (BDVA)

The BDVA is an industry-driven and fully self-financed international not-for-profit organisation under Belgian law. The BDVA has over 220 members all over Europe with a well-balanced composition of large, small and medium-sized industries (over 30% of SMEs), as well as research and user organisations. The Big Data Value Association is the private counterpart to the European Commission in implementing the BDV PPP.

BDVA members come together to collaborate on a joint mission: developing the European Big Data Value Ecosystem (BDVe) that will enable the data-driven digital transformation in Europe, delivering maximum economic and societal benefit, and achieving and sustaining Europe’s leadership on big data value creation and Artificial Intelligence (Zillner et al. 2019). To achieve this mission, in 2017, the BDVA defined four strategic priorities:

• Develop Data Innovation Recommendations: Providing guidelines and recommendations on data innovation to the industry, researchers, markets and policymakers

• Develop Ecosystem: Developing and strengthening the European Big Data Value Ecosystem

• Guiding Standards: Driving Big Data standardisation and interoperability priorities, and influencing standardisation bodies and industrial alliances

• Know-How and Skills: Improving the adoption of Big Data through the exchange of knowledge, skills and best practices

Since 2017 the cross-technological nature of the data value chains, flowing across different technologies (IoT, Cloud, 5G, Cybersecurity, infrastructures, HPC, etc.), has triggered and accelerated the development of stronger collaborations between the BDV PPP/BDVA and other technological (cross-sectorial) sectorial communities and, in particular, other partnerships.

2.3 BDV PPP Objectives

As laid out in the Contractual Arrangement (CA) of the BDV PPP (BDVPPP Contractual Arrangement n.d.), the overarching general objectives are as follows:

• To foster European Big Data technology leadership in terms of job creation and prosperity by creating a Europe-wide technology and application base, and building up the competence and number of European data companies, including start-ups

• To reinforce Europe’s industrial leadership and ability to compete successfully in the global data value solution market by advancing applications which can be converted into new opportunities, so that European businesses secure a 30% market share by 2020

• To enable research and innovation work, including activities related to interoperability and standardisation, and secure the future basis of Big Data Value creation in Europe

• To facilitate the acceleration of business ecosystems and appropriate business models, with a particular focus on SMEs, enforced by a Europe-wide benchmarking of usage, efficiency and benefits

• To provide and support successful solutions for major societal challenges in Europe, for example in the fields of health, energy, transport and the environment, and agriculture

• To demonstrate the value of Big Data for businesses and the public sector and to increase citizens’ acceptance levels by involving them as “prosumers” and accelerating take-up

• To support the application of EU data protection legislation and provide effective, secure mechanisms to ensure its enforcement in the Cloud and for Big Data, thus facilitating its adoption

2.4 BDV PPP Governance

The main governance structure of the BDV PPP (Fig. 1) was prepared and delivered at the beginning of the PPP to provide the framework for collaboration and alignment among all members of the PPP (EC, funded projects, the Association and its members).

Fig. 1

BDV PPP governance structure

The Cooperation Charter⁴ was created by the Association as one of the key governance mechanisms to facilitate cooperation among the BDV PPP actions and the BDVA and has been updated every year accordingly.

The BDVe project (CSA of the BDV PPP) has supported the implementation of the PPP projects governance structure by establishing the BDV PPP Steering Committee (SC) and the Technical Committee (TC). The Steering Committee (SC) provides executive-level steering and advice to ensure effective and efficient coordination and communication between the BDV PPP actions. The Technical Committee (TC) facilitates knowledge exchange and cooperation on the technical aspects, methodology and implementation of the BDV PPP programme. A non-formal Communication Committee was also established to support cooperation in Marketing and Communications.

The Board of Directors⁵ (BoD) of the BDVA is selected by the General Assembly of the Association (2-year mandate) and is in charge of achieving the objectives of the association. It follows the resolutions, instructions and recommendations adopted by the General Assembly.

The Partnership Board (PB) is the monitoring body of the PPP formed by selected directors of the Board of the BDVA, and representatives of the European Commission. The PB meets approximately 1–2 times per year and complements this with regular bi-weekly exchanges of information. The European Commission is represented by DG Connect Directorate G (Unit G1 in particular).

2.5 BDV PPP Monitoring Framework

The BDVA leads the production of the Monitoring Report of the Big Data Value PPP as part of its contractual obligations in the PPP. The work is developed by the BDVA TF2 (impact). Since 2019 and in accordance with the European Commission, the full monitoring report of the partnership will only be submitted every 2 years. The most recent version was delivered in 2019 covering the period from beginning 2018 to beginning 2019. The list of key performance indicators (KPIs) for this PPP, description and target values are defined by the following documents:

• 1. A note released by Directorate-General for Research and Innovation at the European Commission (DG RTD) in February 2018, defining, describing and providing a methodology for the four common KPIs to all the PPPs

• 2. BDV PPP contractual agreement⁶

• 3. BDV PPP SRIA

To produce the monitoring reports the association gathers input from all the running and selected Big Data PPP projects, all for-profit project partners from the projects, the members of the BDVA, the BDVA Task Forces and the BDVA Office, the EC DG CNECT G1 Unit and the European Data Market Monitoring Tool.⁷

3 Main Activities and Achievements During 2018

The main achievements of the Big Data Value PPP during 2018 can be summarised as follows:

• Mobilised private investments since the launch of the PPP (and until end of 2018) of €1.57 billion (€468 million in 2018). Considering the amount of EU funding allocated to the PPP by the end of 2018 (€201.30 million), the BDV PPP ended that period with a leverage factor of 7.8, much higher than the leverage factor of 4 committed contractually.

• Forty-two projects were running at the beginning of January 2019, with 32 projects active during 2018 (contributing to many of the KPIs) and 10 additional projects selected for funding in 2018 and starting in 2019. Participation of SMEs over 20% and targeted data incubation activities for start-ups (25.3% SME participation in the call for proposals 2018).

• The BDV PPP organised 181 training activities involving over 18,300 participants during 2018. Projects have contributed with 85 training activities during 2018 involving over 9700 participants. BDVA members contributed with 96 training activities involving over 8500 participants. Projects have developed 16 interdisciplinary programmes during 2018 reaching 250 participants.

• Forty-eight job profiles identified by projects in 2018.

• Data skills activities, including the launch of the BDV PPP Educational Hub for European MSc programmes in Data Science and Data Analytics and the launch of the pilot on the skills recognition programme

• Organisation of 323 events reaching over 630,000 participants during 2018.

• One hundred and six innovations of exploitable value, 39 of which are significant (37%), delivered by running projects during 2018. The BDVe project launches the Big Data Value Marketplace⁸ and the Big Data Value Landscape.⁹

• Seventy-seven per cent of the BDV PPP projects contributing to job creation by 2023, with an estimation in accumulated numbers of thousands. Estimated numbers surpass 7500 new jobs created by 2023 linked to project activities and much more considering indirect effect.

• Two patents, over 61 publications and 24 products or software components in the field of advanced privacy- and security-respecting solutions for data access, processing and analysis in 2018.

• Sixty-three new economically viable services of high societal value developed during 2018.

• 100% coverage of research priorities defined in SRIA, with 204 new systems and technologies developed in different sectors during 2018. The major focus of technical contributions lies in “Data Analytics”.

• Two hundred and twenty-four use cases and/or experiments conducted during 2018 by projects and 165 additional experiments conducted by BDVA i-Spaces.

• Eighty-two large-scale experiments were developed by the projects during 2018, 64 involving closed (private) data. BDVA i-Spaces also contributed to this KPI, reporting in total 38 large-scale experiments performed during 2018, 28 involving private data.

• Four major sectors (bio-economy; transport, mobility and logistics; healthcare; smart manufacturing) covered with close-to-market large-scale implementations, and over 15 different sectors covered in total including (in addition to the ones already mentioned) telecom, Earth observation, media, retail, energy, finance and banking, public services, water and natural resources, business services, smart cities insurance, public safety, personal security, public tenders, e-commerce, marketing, fashion industry, citizen engagement, ICT/Cloud services, social networks, procurement and legal services.

• 0,10696 Exabytes (106.73 Petabytes) of data made available for experimentation (86,25 Petabytes by projects, 20,71 Petabytes by i-Spaces).

• Evidence of contribution to the environmental KPIs, with some pilots showing 25% and 51% in energy reduction and improvements concerning CO₂ emissions, reaching up to 29% and 23% of emission reductions in general.

• “During 2018, 396 FTEs master and PhD students (60 masters and 136 PhD) were involved in PPP projects”.

• SME turnover evolution increase of 60% with respect to 2014 and 17.7% in the last year. A positive trend in employment evolution with an average increase in employment for the SMEs that are part of the PPP is 75% with respect to 2014 and a growth of 11.83% in the last year.

• The European Data Incubators DataPitch and EDI have given support and new opportunities to 47 start-ups and entrepreneurs, creating an impact on revenues, jobs created and competitiveness, and supporting them to raise additional private finding.

• Third wave of BDVA i-Spaces labelling,¹⁰ with 10 labelled i-Spaces selected during 2018.

• The BDVA joined the EuroHPC Joint Undertaking¹¹ as one of its private members, bringing synergy between Big Data, AI and HPC for industrial use-cases. The BDVA appoints two official representatives in the EuroHPC RIAG and two additional observers.

• During 2018 the BDVA developed collaborations with impact on technology integration, roadmapping and the digitisation of industry challenges.

• The BDVA delivered 7 strategic papers during 2018 supporting this strategic roadmap, and an additional 10 new papers were released in 2019 and early 2020, including on essential topics such as data protection in the era of Artificial Intelligence and use of data in Smart Manufacturing.

• Official liaison with the ISO/JTC1/SC42 with the main objective of channelling European input (PPP) into global standards for AI and Big Data. In 2019 the BDVA was in the process to sign an MoU with CEN (European Committee for Standardization)/CENELEC (European Committee for Electromechanical Standardization) and ETSI (European Standards Telecommunications Institute).

• During 2018 the BDVA also developed strong foundations for the future, building upon the current BDV PPP by joining the EuroHPC Joint Undertaking (as a private member) and by driving (together with euRobotics) the future partnership on AI, data and Robotics. As of today, the BDVA is the main promoter of the AI, Data and Robotics partnership,¹² one of the candidates for European Partnerships in digital, industry and space in Horizon Europe, developed in collaboration with euRobotics and the AI Research communities CLAIRE, ELLIS and EurAI.

• The BDVe project has supported the collaboration of the Network of Centres of Excellence in Big Data and the establishment of a new Centre of Excellence in Bulgaria, the first such centre in Eastern Europe.

3.1 Mobilisation of Stakeholders, Outreach, Success Stories

The year 2018 was one of remarkable progress and advancements for the Big Data Value PPP and the BDVA. In its second year of operations, the PPP showed a great quantity and variety of success stories from projects and the association. The main success stories from the projects related to:

• The impact created in specific sectors (e.g. results in the Lighthouse projects TT and DataBio already reporting evidence on reduction of operation and production costs, reduction of emissions, improvements on energy efficiency, etc.)

• “Close to market” technology and solutions delivered (e.g. FLAIR (framework for Natural Language Processing developed by FashionBrain) already integrated into the PyTorch ecosystem, or SLIPO workbench already used by other PPP projects and in commercial settings in the PPP Point of Interest (POI) data sets on a global scale)

• Performance (e.g. in one of its pilots BigMedilytics achieved a better prediction of re-admission for chronic heart failure over 50%)

• Resources generated (new knowledge, new ontologies, datasets)

• Incubation of new data-driven businesses (47 start-ups in 2018 with individual success stories)

• Research excellence (publications and paper awards)

• Impact in Standardisation

• Strong foundations put in place for future activities

The European Data Incubators/accelerators DataPitch and EDI gave support and new opportunities to 47 start-ups and entrepreneurs, helping them to grow their business in the new Data Economy offering skills development, access to resources, data, infrastructure, ecosystem and additional private funding. This has generated a significant impact on revenues, jobs created and competitiveness.

It is important to highlight the positive effect that participation in a more extensive programme has brought to individual projects. Eighty per cent of the projects reported value created for their Research and Innovation projects by being part of the BDV PPP, e.g. facilitating collaboration and exchanges between projects, such as complementary functionalities (e.g. SLIPO and QROWD), reuse of projects outcomes (functionality, solutions or ontologies, data sharing¹³ and specific know-how sharing). Additionally, the PPP is seeking to be effective in coordinating communication activities, providing new opportunities for start-ups, and providing a common framework and vocabulary to develop effective end-to-end ecosystems.

It is also quite remarkable to note the overall impact in communication and engagement of the PPP, with the estimated number of people outreached in dissemination activities around 7.8 million in 2018 with the objective of raising awareness about their different activities, to engage new stakeholders, and communicating the result. Additionally, the BDV PPP organised 181 training activities involving over 18,300 participants during 2018. The range and diversity of actors and stakeholders outreached is very broad, in alignment with the overall objectives of the PPP.

4 Monitored Achievements and Impact of the PPP

Enabled by the monitoring framework, as described above, the progress of the BDV PPP is continuously monitored. Below we report the key achievements and impacts in alignment with the development phases described in the SRIA that are backed by the monitoring data.

4.1 Achievement of the Goals of the PPP

According to the Big Data Value PPP SRIA v4,¹⁴ the programme would develop the European data ecosystem in three distinct phases of development, each with a primary theme:

• Phase I: Establish the ecosystem (governance, i-Space, education, enablers) and demonstrate the value of existing technology in high-impact sectors (Lighthouses, technical projects) (Work Programme WP 16–17)

• Phase II: Pioneer disruptive new forms of Big Data Value solutions (Lighthouses, technical projects) in high-impact domains of importance for EU industry, addressing emerging challenges of the data economy (WP 18–19)

• Phase III: Develop long-term ecosystem enablers to maximise sustainability for economic and societal benefit (WP 19–20)

The PPP goals achieved are analysed based on the defined roadmap. The year 2018 lies between Phase I and Phase II, and thus the progress of the PPP is assessed considering the objectives of both phases.

Phase I: Establish an Innovation Ecosystem (WP 2016–17) focused on laying the foundations needed to establish a sustainable European data innovation ecosystem (Table 1).

Table 1 Summary achievements of the goals of the BDVA PPP: Phase I of the roadmap

Phase II: Pioneer disruptive new forms of Big Data Value solutions (Lighthouses, technical projects) in high-impact domains of importance for EU industry, addressing emerging challenges of the data economy (WP 18–19). According to the SRIA, this second phase is meant to build on the foundations established in Phase I and will have a primary focus on Research and Innovation (R&I) activities to deliver the next generation of Big Data Value solutions. Although the projects implementing Phase II started in 2019 (or 2020), there are some activities in 2018 supporting the implementation of this stage, in particular those listed in Table 2.

Table 2 Summary of achievements of the goals of the BDVA PPP: Phase II of the roadmap

Phase III¹⁵: Develop long-term ecosystem enablers to maximise sustainability for economic and societal benefit (WP 19–20). This phase started in late 2019 and will continue until the end of the PPP. As this phase has only just started, the analysis can only be incomplete. Some ideas about possible achievements are provided in Table 3.

Table 3 Summary of achievements of the goals of the BDVA PPP: Phase III of the roadmap

4.2 Progress Achieved on KPIs

4.2.1 Private Investments

Through this KPI, we attempt to understand and capture/show the level of industrial engagement within the BDV PPP. This KPI includes both direct and indirect leverage, as described in Fig. 2.

Fig. 2

Methodology and KPI structure proposed by EC for MR2018 (PPPs) (by European Commission licensed under CC BY 4.0)

Two hundred and ninety-six companies representing all for-profit organisations participating in Big Data Value PPP projects active during 2018 (including not only project partners but also third parties engaged through cascade funding) and all for-profit organisation members of the BDVA were outreached to provide input to this KPI with an overall response rate of 40.9%.

Table 4 shows the evolution of the reported numbers in private investments from 2015 to 2018, as well as the EU contributions.

Table 4 Evolution of private investments in BDV PPP over time

Aggregated to the numbers reported in 2015 (€280.9 million), 2016 (€338.5 million) and 2017 (€482.25 million), the amount of mobilised private investments since the launch of the PPP until the end of 2018 was 1569.1M€ (€1.57 billion). Considering the amount of EU funding allocated to the PPP by that time (€201.3 million), the BDV PPP ended 2018 with a leverage factor of 7.8, much higher than the leverage factor of 4 committed contractually.

4.2.2 Job Creation, New Skills and Job Profiles

Seventy-seven per cent of the BDV PPP projects indicated that their project would contribute to job creation by 2023, with an estimation in accumulated numbers of thousands. The estimated numbers surpass 7500 new jobs created by 2023 linked to project activities and many more considering indirect effect.

BDV PPP projects contribute to job creation in Europe by (1) increasing the market share of Big Data Technology providers in Europe; (2) developing new job profiles that generate new jobs… the creation; (3) developing new opportunities for entrepreneurs and start-ups in the new Data Economy; (4) generating job opportunities by increasing data sharing; (5) creating new jobs already during the lifetime of the project; and (6) forecasting jobs created as a follow-up of project results.

On the other hand, 40% of the BDVA members stated that their participation in the BDVA/BDV PPP had already contributed directly or indirectly to job creation, mainly because of the hiring of new experts to develop H2020 projects, start-ups created...), and new profiles hired to develop operations.

Projects reported that 48 job profiles were created or identified in 2018, and 106 new job profiles were reported as expected to be created from 2019 onwards and by the end of the project linked to the project activities.

Sixty-seven per cent of the projects running in 2018 reported contribution to the generation of new skills by the end of the project. In addition to the skills linked to the new job profiles, new skills are expected to be developed in cross-sectorial domains (e.g. in the form of “privacy-aware data processing” and “privacy-aware big data innovation” as reported by the SPECIAL project) and in specific sectors (e.g. analysis techniques using weather data, reported by the EW-SHOPP project). The BDV PPP incubators help start-ups to develop both the technical and non-technical skills needed to develop business in the Data Economy.

Among BDVA members, 51% of organisations reported contribution to the creation of new job profiles, and almost 60% contribute to the creation of new skills linked to the Big Data Value PPP. Finally, 60% of the projects and 51% of BDVA members have reported contributions to the Skills Agenda for Europe.

The BDV PPP organised 181 training activities involving over 18,300 participants during 2018. Projects contributed to this with 85 training activities during 2018 involving over 9700 participants. BDVA members reported 96 training activities involving over 8500 participants. Projects developed 16 interdisciplinary programmes during 2018 outreaching 250 participants.

During 2018, 396 equivalent FTEs masters and PhD students “(260 masters and 136 PhD) were involved in PPP projects, thereby collaborating with industrial players in developing industry-driven solutions and deploying experimentation testing scenarios. Contributing to raising awareness in professionals, users and the general public, the BDV PPP organised 323 events outreaching around 630,000 participants during 2018 contributing to raising awareness in professionals, users and the general public.

4.2.3 Impact of the BDV PPP on SMEs

Results of the Monitoring Report 2018 showed that a wide range of SMEs in Europe benefit from the Big Data Value PPP, considering the size (12% medium-size companies, 41% small companies and 48% micro-companies¹⁶), age (20% of the SMEs are 0 to 4 years old, 36% are 5 to 10 years old and 42% are 10 years old or older) and wide geographical distribution. SMEs play a variety of roles in the data value chain. SMEs participating in PPP projects clearly show a trend of an increase in turnover and in the number of employees. It is also important to mention that not all the SMEs involved in BDV PPP projects are technology companies but are also data users or providers, and the overall results and trend indicate an ongoing growth of turnover along the whole value chain.

Total turnover reported for SMEs in 2017 was €260.4 million.¹⁷ In terms of turnover evolution, there is an increase in turnover in the SME companies that are part of the PPP with reported numbers of 60% increase in turnover with respect to 2014 and 17.7% in the last year. This number is in full alignment with the macro-economic numbers of data companies in Europe, and higher for some specific categories. In particular, young SMEs (5 and 10 years old) show on average the highest growth in turnover in relation to 2014 (up to 284%). The youngest companies (<5 years) show on average the highest growth in the last year (54.8%).

In terms of employment evolution, the trend is also very positive in all companies that are part of the PPP, with an average increase in employment for the SMEs that are part of the PPP of 75% with respect to 2014 and a growth of 11.83% in the last year (2018 compared with 2017).

Special emphasis should be given to PPP instruments focused on supporting SMEs, in particular the Data Incubators and i-Spaces. The average age of the companies receiving cascade funding from the Data Incubators (DataPitch and EDI) is 4.9 years; 41% of those SMEs are younger than 5 years, 50% are between 5 and 10 years, and only 9% are older than 10 years old. Companies reported an increase in turnover of 315% for 2014 and 48.8% for 2017, and an 118.5% increase in employment for 2014 with a 22.4% increase in the last year.

4.2.4 Innovations Emerging from Projects

Innovations arising from the BDV PPP include:

• Specific project developments that have a marketable value, including Big Data products, processes, instruments, methods, systems and technologies, offering value to a wide variety of economic and industrial sectors

• Services of high societal value developed by projects

• Spin-offs arising from projects and start-ups incubated by the programme activities

• Patents and solutions enabling advanced privacy- and security-respecting solutions for data access, processing and analysis

• Contribution to Standards (individually as projects and coordinated activities at a programme level)

• Innovations resulting from cooperation between projects or programme-coordinated activities (e.g. advances in data sharing, innovative skill programmes, reuse of technical solutions across different sectors, etc.)

• Transformation of sectors of high economic value (led by the PPP Lighthouse projects, but also triggered by project cooperation): new business models and scaling innovations (advances in TRLs (technology readiness levels), cross-border solutions and bringing technology closer to the market, accelerating adoption)

In its second year of operation, the BDV PPP’s 32 running projects reported 106 innovations of exploitable value as delivered in 2018: 63% have a medium impact and 37% are considered innovations of significant impact. Fifty per cent of the innovations delivered in 2018 are incremental innovations, 6% are architectural, 36% are disruptive and 1% are radical innovations.¹⁸

Ninety-three per cent of the innovations delivered in 2018 have an economic impact, and 48% have a societal impact.¹⁹ Forty-one per cent are technologies (including platforms), 32% are services, 7% are products, 8% are methods, 8% are systems, 1% are software, 4% are components and/or modules and 11% are others, including frameworks/architectures, processes, tools and toolkits, spin-offs, datasets, ontologies, patents and knowledge.

Sixteen per cent of the innovations delivered in 2018 are fully cross-sectorial.

Sevety-five per cent provide solutions to the transport, mobility and logistics sector (the one with the best coverage in the PPP by the end of 2018); 20% of the innovations related to public services and smart cities; 19% to industry and manufacturing; 14% to bio-economy; 13% to the Telco sector; 12% marketing activities; 8% relate to health and healthcare; 8% to the ICT market; 7% to geospatial market; 5% to commerce; and 3% to others (including fashion, retail, business services, energy, media, compliance, etc.).

In relation to the maturity levels and TRLs, 7% of the innovations delivered are TRL 3 (experimental proof of concept), 10% are TRL 4 (technology validated in lab), 36% are TRL 5 or TRL 6 (technology validated in relevant environment, industrially relevant environment in the case of key enabling technologies), 32% are TRL 7 (system prototype demonstration in operational environment), 8% are TRL 8 (system complete and qualified) and 1% are TRL 9 (actual system proven in an operational environment—competitive manufacturing in the case of key enabling technologies—or in space).

Figure 3 provides a full overview of the innovations delivered by the BDV PPP during 2018, combining level of significance, type of innovation (incremental, disruptive, architectural or radical) and the TRLs. Although a large number of innovations are classified as incremental innovations of medium impact, it is remarkable to note the high percentage of significant innovations (and expected growth in the upcoming years), the high number of disruptive innovations and the high TRLs in some cases close to deployment. Although at a lower level, the BDV PPP is also delivering some architectural and radical innovations.

Fig. 3

BDV PPP innovations to market 2018

Sixty-three new economically viable services of high societal value were developed during 2018 as a result of the projects. Forty-seven per cent (over 30 projects) contributed to this KPI.

Projects reported 204 new systems and technologies developed during 2018. Many of them are already reported as part of the KPI “Significant Innovations to Market”. Systems and technologies developed are not limited to one sector, and, in fact, the majority of the new systems and technologies can be utilised in different sectors/markets, thus stimulating the use of Big Data technologies in many areas.

Finally, many solutions and innovations arising from the Big Data PPP have been promoted in the BDV PPP Marketplace²⁰ developed by the BDVe CSA project to spread knowledge about the outcomes of the PPP.

4.2.5 Supporting Major Sectors and Major Domains with Big Data Technologies and Applications

The BDV PPP Lighthouse projects²¹ active in 2018 focused on the bio-economy (agriculture, fisheries and forestry) (DataBio project), transport, mobility and logistics (transforming transport project), health and healthcare (BigMedilytics project) and manufacturing (BOOST4.0), with a total of four major sectors supported by Lighthouse projects and therefore widely supported by multiple use cases, scenarios and solutions.

Twenty per cent of the projects are fully cross-sectorial (their outcomes can be used in any sector or application domain) and 80% of the projects are working in more than one sector or application domain (this explains why the total is higher than 100% in Table 5). In particular, the BDV PPP projects address a wide variety of sectors²², as shown in Table 5.

Table 5 Support to major sectors and domains

Others (43% of the projects) includes sectors such as insurance, public safety, personal security, public tenders, e-commerce, marketing, fashion industry, citizen engagement, ICT/Cloud services, social networks, procurement and legal domain.

Considering the whole project portfolio, the number of sectors supported is higher than 15, with a solid distribution of use cases, experiments, solutions and outreach activities among different sectors.

4.2.6 Experimentation

Projects reported 224 use cases and/or experiments conducted during 2018 with contributions from 18 different projects. This is an increase of 48.3% with respect to 2017 (151 experiments). The BDVA i-Spaces reported an additional 165 experiments with 6 i-Spaces contributing to this KPI.

Projects reported 82 large-scale experiments developed during 2018, 64 involving closed (private) data (78% of the total). Large-scale experiments either involve a large number of users with high TRLs or are developed in large geographical areas, in many cases involving a large number of users and actors or a combination of data volume, complexity and velocity; a large number of data sources; or integrated complex datasets flowing across borders. The BDVA i-Spaces also contributed to this KPI, reporting in total 38 large-scale experiments performed during 2018, 28 of them involving private data.

In relation to the amount of data made available for experimentation, reported information from projects and i-Spaces (members of BDVA) shows that the amount of data made available by the BDV PPP for experimentation in 2018 is 0.10696 Exabytes (106.96 Petabytes). A total of 0.08625 Exabytes (86.25 Petabytes) was reported by the projects.²³ It is important to note that some of the projects are not only providing internal access to diverse data sets from different sources but are also improving and creating new valuable datasets (e.g. of DataBio project). BDVA i-Spaces contributed to this KPI, reporting an additional 20.71 Petabytes of data for experimentation.

4.2.7 SRIA Implementation and Update

Concerning SRIA coverage, measured as “% of research priorities covered compared to the overall scope of research priorities defined in SRIA”, projects have delivered contributions during 2018 already covering 100% of all the SRIA technical priorities. The major focus of technical contributions was “Data Analytics”, followed at some distance by “Data Processing Architectures” and “Data Management”. This is a significant change from the 2017 coverage, where “Data Management” was the top priority. A clear trend to focus on technical contributions in the areas of “Data Analytics” and “Data Processing Architectures” was anticipated in the BDV PPP Annual Monitoring Report 2017,²⁴ thus supporting our explanation that a solid base of “Data Management” solutions will enable analytics and processing innovations.

In relation to the BDV SRIA update, at the end of 2017 the BDVA released the BDV PPP SRIA v4.0 (detailed process and results reported in the 2017 Monitoring Report). This version was the basis to support the H2020 LEIT ICT WP2018–20. During 2018 a minor update, towards a version 4.1, was released in the community, crystallising in a series of individual deliverables in the format of vision, position or discussion papers that supported the transition towards the next framework programme and the creation of a new strategic agenda and roadmap.

In total, there were at least 12 events organised during 2018 that contributed to input in the BDVA strategic papers – multiple online meetings with a total of 2085 participants/contributions.

In total, since the launch of the BDV PPP, we can count 6422 potential contributions to the strategic roadmapping activities.

4.2.8 Technical Projects

The BDV PPP contributes to enabling advanced privacy- and security-respecting solutions for data access, processing and analysis. For 2018, 97 contributions were reported (2 patents,²⁵ 61 publications and 24 OSS/SW/Products).

Fifty per cent of the projects confirmed that they are assessing quality, diversity and value of data assets. These results show the intense usage of metrics to measure quality, diversity and value of data assets in projects, and some projects have developed specific metrics and methods to ensure quality, diversity and value in the data (e.g. I-BiDaaS has developed a Data Quality Assurance Process (DQAP) aiming at ensuring the high quality of the data generated/collected during the lifetime of the project). However, we cannot talk yet (2018) about the “PPP”-developed metric expected for 2019+.

Concerning the speed of data throughput, 40% of the projects reported that they expect the project to improve data throughput. Some projects, such as BigDataOcean and FashionBrain, measured improvements over 1000%. Others such as I-BiDaaS have specific objectives to develop data processing tools and techniques applicable in real-world settings and to demonstrate a significant increase in speed of data throughput and access.

4.2.9 Macro-economic KPIs

The monitoring of macro-economic KPIs is based on input from the European Data Market Monitoring Tool²⁶ as they are presented in the most recent report by IDC (https://www.idc.com/).²⁷

Development of the market share of the European Union in the global Big Data Market. As an indicator, we compare the total revenues of EU Data Companies with other economies, i.e. the US, Japan and Brazil, as they are used as a benchmark in the IDC report.²⁸ The EU share of the total revenues in these economies the 2013 baseline was 27.7%. This share increased slightly to 27.9% in 2018, which is remarkable because the international indicators grew very fast in this period, but the EU kept pace with them. In absolute terms, the total revenue of US data companies in 2018 was approximately twice that of EU28 data companies in the same year (€162 billion vs. €77 billion). Seventy per cent of PPP projects active in 2018²⁹ reported contribution to increasing the revenue share of EU companies. Projects contributed by:

• Accelerating adoption of new technologies

• Supporting EU data-driven companies to build innovative solutions that can be scaled internationally

• Developing innovative technologies to make European companies more competitive (e.g. news data protection approaches)

• Enabling industries to exploit their big data efficiently and therefore increase their market share and services provided to their customers

According to the most recent report,³⁰ the number of data companies increased to 283,100 by 2018, compared to 271,700 in 2017, with a growth rate of 4.2%. It should be noted that almost half of them are based in the UK, due to the high concentration of the ICT industry there. BDVA i-Spaces and Data Incubators (ICT 14-b projects, i.e. DataPitchand EDI projects) are in particular designed to contribute to this KPI as they support start-ups and entrepreneurs from early ideas to technical and business development until the go-to-market stage.³¹ Seventy-seven per cent of the BDV PPP projects active in 2018³² reported contribution towards increasing the number of European companies offering data technology and applications. The projects contributed in different ways, such as:

• Creating tools that will stimulate the creation of new companies

• Creating new companies as a result of a project (e.g. BigDataOcean)

• Supporting EU data-driven companies

• Building innovative solutions to solve data-related challenges

• Supporting companies in complying with the GDPR

• Lowering the threshold to create new business in a particular sector

In addition, 25% of BDVA members reported that their organisation ran or supported a programme that is specifically targeted at supporting start-ups or entrepreneurs in the field of Big Data.

The revenue of data companies in the European Union, according to the IDC report,³³ reached €77 billion in 2018 compared to €69 billion the previous year, with a growth rate of 12%. The revenue share of SMEs in 2018 amounts to €55.5 billion (72% of the total revenue), an absolute growth of €5.7 billion on the year before. The growth rate of revenue increases in proportion to company size, with the revenue of large companies with over 500 employees growing at 16% in 2018 over 2017. Seventy-seven per cent of the PPP projects active in 2018³⁴ reported contribution (or plan to contribute) to the revenue generated by European data companies. Project contribution to this KPI is mainly by:

• Opening up sectors to data-intensive companies

• Offering direct support and getting funding for data start-ups

• Making data processing easier and cheaper for companies

• Creating new opportunities through privacy-preserving analytics solutions

• Commercialising new services with a marketable value

• Creating opportunities for common exploitation based on joint Big Data technology pipelines

• Developing simplicity in some business ecosystems

The baseline for data professionals in the European Union in 2013 amounts to 5.77 million. The number of data professionals increased to a total of 7.2 million by 2018, resulting in an absolute growth rate of 1453 million professionals since 2013. The rate of growth of data professionals is increasing, with approximately 559,000 positions added in 2018 and an increase of 8.4% on the year before.³⁵ Eighty-seven per cent of the PPP projects active in 2018³⁶ reported contribution from their project to increase the number of data workers in Europe. Projects contribute to this KPI in different ways:

• New organisations created as a result of the projects hiring new data professionals

• Supporting growth of emerging start-ups

• Developing more data-driven services that will require new data workers

• Unlocking the value of data services by introducing privacy-preserving technologies

• Creating new job profiles

• Supporting the adoption of data solutions in different sectors

• Supporting education and training

4.2.10 Contributions to Environmental Challenges

Over 20% of the projects running in 2018 reported that they contribute to the reduction of energy, and 30% contribute to reduction in CO₂ emission. Quantitative results are provided by some projects, such as the Transforming Transport (TT) project that shows that in some specific monitored items improvements in efficiency range between 25% and 51% in energy reduction, and improvements concerning CO₂ emissions reach up to 29% and emission reductions in general (including PM and NOx) up to 23%.

The three Lighthouse projects running in 2018 (DataBio, Transforming Transport and Boost4.0) have reported contribution to reduction in waste. For example, in DataBio and in particular in forestry, although still with early data and experiments, the experience from customer cases shows a reduction in waste of up to 10%. Some pilot TT projects show approximately 25% improvement in the management of assets, which can adequately demonstrate a relative high-level achievement in waste reduction at this final stage of the project.

Seventeen per cent of the projects running in 2018 have reported contribution to reduction in the use of material resources; e.g. BigMedilytics provides quantitative data in a particular scenario, reporting that the Asset Management pilot aims to reduce the number of unused mobile assets in hospitals by up to 20%.

Finally, in relation to energy reduction in big data analytics, there is no quantitative input in results provided by any project but, e.g., the E2Data project develops a framework that optimises calculations, leading to decreased use of energy.

4.2.11 Standardisation Activities with European Standardisation Bodies

During 2017, the BDVA and the BDV PPP set up some foundations defining priorities for the PPP in Big Data standardisation implemented during 2018 as follows:

• Establish an official liaison between the BDVA Standards Group and the AIOTI WG3; this activity was developed through different workshops during 2017 and implemented in 2018 with the signing of an MoU with AIOTI and common activities organised during the year.

• Further develop the BDVA Reference Model pursuing alignment with others, such as oneM2M, BDE Platform, AIOTI and RAMI 4.0, implemented through different workshops organised during 2017 and 2018.

• Open an official dialogue with CEN, CENELEC and ETSI on standards harmonisation, implemented through different workshops during 2017 and 2018. The BDVA intends to sign an MoU with CEN/CENELEC in 2019, and it is under discussion with ETSI.

• Create the BDVA Roadmap for Big Data standards harmonisation and industry engagement in Global Big Data standards development.

Thirty per cent of the projects running in 2018 reported that they perform activities leading to data/Big data standardisation. Three projects reported contribution to European standardisation bodies (ESBs) activities and reported 11 working items in ESBs. Twenty per cent of BDVA members reported that their organisations perform activities leading to data/Big data standardisation. In particular, BDVA members have reported contributions to IEC, DIN DKE and other consensus-based standardisation bodies; OPC foundation and other consortia-based standardisation bodies; OASIS; W3C committees and community group discussions; open data harmonisation national activities; ISO/IEC JTC1; and defining standards in georeferenced data for geoscience (Open Geospatial Consortium (OGC) and Commission for the Management and Application of Geoscience Information (IUGS/CGI)) and ETSI.

5 Summary and Outlook

The year 2018 was a transition year and an important inflexion point between the so-called Phase I (establishment of the ecosystem) and Phase II of the BDV PPP (pioneer disruptive new forms of big data value solutions). New calls for proposals were in place during 2018 and 2019 as part of the H2020 WP 2018–2020 (calls closing in April 2018, November 2018, April 2019 and November 2019) that brought new projects that enriched the BDV PPP portfolio, also increasing challenges of coordination, communication and cooperation. The year 2018 was also a transition year in defining the strategy and direction of the next partnership framework programme (2021–2028).

The increase in the quality and quantity of the data available for experimentation and the launch of the cross-border Industrial Data Platforms and Personal Data Platforms at the beginning of 2020, supported by other ecosystem enablers, have directed the final transition period towards Phase III as defined in SRIA v4. The BDV PPP projects starting in 2020 (e.g. EUHubs4Data project) are establishing a strong foundation for the next framework programme (deployment of data platforms, the federation of Big Data Innovation Hubs/data experimentation facilities, and advances in data and data-driven AI capabilities).

On 25 April 2018, the European Commission outlined a European strategy for AI to boost investment and set ethical guidelines. In its communication, the European Commission put forward a European approach to Artificial Intelligence based on three pillars: (i) “boosting financial support and encouraging uptake by public and private sectors”, (ii) “preparing for socio-economic changes brought about by AI”, and (iii) “ensuring an appropriate ethical and legal framework”. The strategy acknowledged that member states had existing research and innovation objectives that focused on AI and encouraged alignment of individual roadmaps towards a European partnership. Also on 25 April the European Commission proposed a package of measures as a key step towards a common data space in the EU – a seamless digital area with a scale that will enable the development of new products and services based on data.

On 6 June 2018, the European Commission announced its proposal to create the first ever Digital Europe programme and invest €9.2 billion to align the next long-term EU budget 2021–2027 with increasing digital challenges. The Commission’s proposal focused on five areas: supercomputers, Artificial Intelligence (AI) (including Data/European Data Space), cybersecurity and trust, digital skills, and ensuring a wide use of digital technologies across the economy and society.

On 7 June 2018, the European Commission announced Horizon Europe (research and innovation programme for the next long-term EU budget 2021–2027) with plans to bring a new generation of European Partnerships and increase collaboration with other EU programmes.

Towards the end of 2018, the BDVA committed its official participation as a private member of the EuroHPC Joint Undertaking aiming at bringing synergy between HPC, Big Data and Artificial Intelligence, and providing industry perspective.

Additionally, the BDVA and euRobotics officially joined forces at the end of 2018 and announced their intentions of working together in a future AI, Data and Robotics Partnership. On 7 December 2018, the European Commission presented a coordinated plan prepared with the members states to foster the development and use of AI in Europe. The plan proposes the development of a European AI public-private partnership building on the BDV PPP and SPARC PPPs.

During 2019 the BDVA and euRobotics developed a common vision paper and the first version of a common AI-PPP Strategic, Research Innovation and Deployment Agenda with strong involvement of ongoing PPP projects, members and many external communities. At the end of 2019, CLAIRE, ELLIS and EurAI joined forces with the BDVA and euRobotics, and the five organisations submitted a joint Partnership Proposal (Zillner et al. 2020). This document lays down the context, vision and objective, and suggests the impact of a possible Partnership of Data, AI and Robotics, building upon the strong assets developed by the BDV PPP and the SPARC PPP. During the first months of 2020, the member states and the European Commission carefully considered the Partnership Proposal and provided feedback for its improvement, which resulted in several updates of the document. On 22 September 2020, the joint release of the Strategic Research and Deployment Agenda (SRIDA v3.0) was published, paving the way towards the new Partnership for Horizon Europe and the Digital Europe Programme, bringing investments and new instruments to scale up the assets and impact of the current Big Data Value PPP.