Perceptions of Final Beneficiaries about the Performance of Cross-sector Partnerships: A Case Study Applied to the 2008 Zaragoza International Exhibition on Water and Sustainable Development

Abstract

Using the 2008 Zaragoza International Exhibition “Water and sustainable development” as a case study, this paper aims to respond to the increasing demand for measurements of the effects and the implications of the performance of cross-sector partnerships from the perspective of their intended final beneficiaries. A contingency framework for measuring the short-, medium- and long-term effects of the 2008 Zaragoza International Exhibition is developed based on a “results chain” or “logic model”. Our results highlight that there are positive long-term synergies between the two main purposes of the 2008 Zaragoza International Exhibition; first, to increase public awareness of and commitment to the problems of water and sustainable development and, second, to make the city of Zaragoza better known internationally and to modernize its infrastructures. Although respondents to our survey consider that the long-term effects on the city are greater, the main short- and medium-term effects are related to awareness of water problems, sustainable development and non-governmental organizations. These results are in tune with what has happened around the city in the last 10 years providing indirect validity both to our study and to the proposed methodology.

1. Introduction

Organizations cannot individually address the complex challenges of sustainability on their own. Joint efforts are needed to integrate environmental, social and economic considerations to facilitate the transition to a more sustainable society [1], to provide social goods such as clean water, health or education [2], to address complex social problems such as poverty and inequality [3] or to fill institutional voids [4]. In order to increase the efficiency and effectiveness of these activities, in recent years, public–private–nonprofit partnerships are growing in number around the world. These cross-sector partnerships have been constituted as organizational solutions to these complex societal problems that call for the comparative advantages of different sectors [5]. The central aim of these partnerships between firms, governmental agencies, non-governmental organizations (NGOs), universities and other organizations is to address these problems through collaboration and the combination of their organizational capacities and resources [6,7,8] to obtain benefits for the wider community rather than for special interests [9].

These partnerships are increasingly being adopted by many companies, which have appreciated their potential to contribute to long-term competitive advantage [10], by governments, which see them as ways of producing public goods in collaboration with NGOs [5] and firms [11], and by many civil society organizations in order to develop novel solutions to complex, old problems [12]. Simultaneously, these partnerships have to address the challenges of assessing and reporting their non-financial performance under increasing demands for measurements of effects and their implications for general performance measurement [3]. It is for this reason that the evaluation of anticipated outputs (short-term effects), outcomes (medium-term effects) and impacts (long-term effects) [13], as well as the “accountability” [14] and the assessments of the efficiency and effectiveness of partnerships in addressing their intended goals are the most critical elements in partnership decisions [15]. As a consequence, there has been an increase in the management literature on cross-sector partnerships [16] and a large number of studies seem to indicate that joint efforts and collaborative approaches can help to build sustainability-oriented organizations [17,18]. However, both theoretical and practical research need more thorough evidence of the effects of cross-sector partnerships and to establish the conditions under which these effects can be enhanced [15].

Ebrahim and Rangan [3] argue that organizations which have a linear theory of change and a tightly-focused operational strategy have the ability to measure their inputs, activities, outputs and outcomes, but they are unable to reasonably measure their impacts. These authors define these impacts as lasting changes in the lives of people and their societies and develop a tool that reasonably measures and explains them [14]. To that end, Ebrahim and Rangan [14] and Van Tulder et al. [15] develop a contingency framework for measuring outputs, outcomes and impacts and their implications for the efficiency and effectiveness of cross-sector partnerships through the use of a logical chain of the social performance of cross-sector partnerships.

Following the literature mentioned above [6,7,8,9], for which the objective of cross-sector partnerships is to help to solve complex problems through collaboration and obtain benefits for the community in general, partnerships management is a relevant matter and the use of performance assessment systems may become relevant to improve efficiency [19]. One way to carry out this assessment is to know the perceptions of the final beneficiaries about the performance of the cross-sector partnerships. With this purpose the authors propose for the first time in the literature, a methodology based on the logic model framework of Van Tulder et al. [15] (see Figure 1 below) and the use of structural equation models to evaluate the performance of inter-sectoral partnerships from the perceptions of final beneficiaries. The procedure is applied to the assessment of the outputs, outcomes and impacts of the 2008 Zaragoza International Exhibition “Water and sustainable development” (EXPO) on people’s awareness of water and sustainable development and of the future development of the city of Zaragoza, which were the main objectives of exhibition. We use the opinions of a sample of Zaragoza citizens who were the final beneficiaries of the EXPO cross-sector partnership in a specifically designed survey.

The study is organized as follows. Section 2 discusses the current state of effects assessments in social performance measurement. Section 3 reviews the theoretical model of a value chain to measure the effects of cross-sector partnerships. Section 4 describes the case study. Section 5 develops a logical model to assess the short-, medium- and long-term effects and the effectiveness and efficiency of the EXPO. Finally, Section 6 concludes. An Appendix, with the equations of the structural model used in the paper, and Supplementary Material, with an additional comparative study by socio-demographic characteristics of the respondents, has also been included.

2. The Current State of Effects Assessments in Social Performance Measurement

In the absence of a universally accepted methodology, partnership researchers are clearly becoming more interested in the objective evaluation of effects and its interaction with the context. In this section we explore the current state of the assessment of effects and how and to what extent this can be achieved.

2.1. The Demand for Evidence-Based Effects Assessments

Cross-sector partnership research is characterized by a growing abundance of methodological approaches to measure the effects of these partnerships [16,20]. However, even though a wide range of assessment models are available, most of them come up against measurement difficulties due to the multi-causality of partnerships [21]. For this reason, there has been very little empirical evaluation, so effects assessment models across sectors remain relatively limited [22]. These circumstances reiterate the importance of moving the discussion on effects towards more concrete evidence-based insights in order to assess when and under which conditions different types of partnerships do or do not work [23]. There is a lack of monitoring and evaluation of the outputs, outcomes and impacts of cross-sector partnerships. That is why governments, NGOs and firms have started to call for better evidence-based effects assessment methodologies [24].

2.2. Concepts and Current State of Effects Assessments in Social Performance Measurement

There is a growing consensus in the literature that “outputs” refer to immediate effects on the participating organizations, while “outcomes” refer to intermediate direct effects on communities or to changes in the lives of individuals, and “impacts” to lasting results achieved at a societal level or to long-term and net effects (direct and indirect effects) on bigger issues [15]. For example, if we wish to reduce poverty through education in a community, we could build a training center. This building would be an input. Then, a certain number of people would attend classes and this number would be an output. As a consequence, more people from that community would have access to specific knowledge of professional training and this would be an outcome. Finally, in the long term, people of that community could work in more qualified jobs and improve their quality of life (this would be an impact).

Nevertheless, the literature on the development of cross-sector partnership initiatives has only identified a limited number of positive impacts for firms, governments and NGOs [25] while it finds numerous positive outcomes and few negative outcomes for individual stakeholders [26]. Furthermore, there is no specific analytical framework for effects assessment that is applicable to all partnerships and such a framework has not been empirically examined in a multi-sector context to date [22] either because, for some complex problems, the long-term effects are still difficult to measure [27] or because little is known about their contribution to wider goals [28]. Gray and Stites [26] suggest that evidence of the effectiveness of multi-sector partnerships still remains largely anecdotal and prescriptive, without clear outcomes of the partnerships on communities and on the environment. In order to make a global assessment of effects, the effectiveness of partnerships needs to be considered in their interaction with the context and their consequences for communities. Most studies conclude that the effects of partnerships need to be addressed at several levels of analysis: community, network, organization and the individuals within participating organizations [22,29]. Critical studies insist on the necessity of investigating this interaction with the development of stronger mechanisms for assessing and ensuring accountability towards both partners and intended beneficiaries if partnerships are to meet their intended objectives [30]. In this paper, we work at community level, focusing on the perceptions of Zaragoza citizens, who we take as representatives of the community that is the final beneficiary of the EXPO.

3. Partnership Monitoring and Evaluation Framework: The Effects Value Chain

Some frameworks for measuring the effects of the social performance of cross-sector partnerships and social sector organizations propose the use of a “results chain” or “logic models” which link the effects of these partnerships to the objectives defined by the participants. These models present a chain of results in which organizational inputs and activities lead to a series of outputs, outcomes and, ultimately, to societal impacts [3]. However, many of these effects are strongly related to broader processes which are difficult to measure and may create indirect and unintended effects that affect the overall results of partnerships. Van Tulder et al. [15] propose an analytical framework for partnership effects assessment which contains two dimensions: (1) a descriptive part that follows an effects value chain and documents the steps of the partnership from issue definition through to impact; (2) an analytical part that covers the four most relevant aspects of partnering (context, efficiency, effectiveness and impact) that assesses the fit and value-added of the partnership to the societal situation. Figure 1 shows the most relevant constituting factors of these two dimensions in the partnership monitoring and evaluation framework.

This effect value chain includes the following elements (see Figure 1):

• The context, which defines the configuration, refers to the particular environment in which the cross-sector partnership operates and can be a country, a region, a city, an issue, or a network. The context conditions and determines the specific factors of the success or failure of partnerships (the degree of efficiency and effectiveness), the nature of the issue and the specific characteristics of the partnership.
• The issue, which can be defined in terms of social problem/opportunity, refers to the social issue being addressed by the partnership. For partners to agree on the social issue they are seeking to tackle is the first step in achieving any kind of effect [31].
• The partners’ mission and goal, which act as the linking pin between the issue and the input, are often defined in a general way and, once the ambition of one party has been achieved, the partnership can be terminated.
• The inputs, i.e., the resources and capabilities provided to achieve the cross-sector partnership’s mission (for example money, staff time, capital assets, commitment), are brought in by each partner (public partners or governments, private partners or firms and community partners or civil society).
• The activities, in whose execution and implementation process partners work towards the partnership objectives.
• The outputs or results of the combination of inputs and activities that a participating organization can measure or assess directly. The output dimension of the effects value chain also includes the benefits for each of the participants in terms of, for example, profits, legitimacy, exposure and moral capital.
• The outcomes are the specific changes in program participants’ (individuals, communities or society at large after) knowledge, status and level of functioning. These changes occur in the short and medium-term and their logical progression should be reflected in impacts occurring in the very long-term in society.
• The impacts are the fundamental and profound changes occurring in organizations, communities or systems as a result of program activities and organizational efforts, including intended and unintended effects and negative and positive effects [32]. Impacts have a long-term character and occur after the conclusion of project funding.
• Efficiency, which refers to the ability to accomplish the partnership activities with a minimum expenditure of time, resources and effort. The efficiency dimension of a partnership can be assessed using a benefit/cost or outputs-outcomes/inputs analysis and it constitutes the internal value added of the partnership. Therefore, the outputs-outcomes/inputs relationship contains two specific dimensions: An operational level of project efficiency that links input with output (G1 in Figure 1) and a tactical level of project performance that links input with outcome (G2). At the operational level of project efficiency, we expect that, once the activities are accomplished, the service delivery will be produced. An example of the operational efficiency of the project “Reducing poverty in a developing country through education” is the ratio of the number of people who attend classes in the new training center to money invested. At the tactical level of project performance, we expect that the accomplishment of these activities will lead to changes of project performance in the short or medium term. An example of a tactical level of performance of the project “Reducing poverty in a developing country through education” is the ratio of number of people who have achieved professional training as a consequence of their studies in the training center to money invested.
• Effectiveness, which is “the value added and the impact of the partnership” compared to the individual activities of the different partners. It measures the quality of the partnership performance in accomplishing the goals and whether the results could have been achieved with a different approach. Like efficiency, effectiveness can also be split into two dimensions: A strategic mission-related performance assessment (H1 in Figure 1) that evaluates how the specific partnership made a difference in context and time, as articulated in the partnership’s mission, and an issue-related performance measure (H2 in Figure 1) that assesses the contribution of the partnership in providing solutions to the social issue initially defined.

Within this analytical framework for partnership assessment, Van Tulder et. [15] propose four impact orders/loops (individual/inside partner, organization, partnership and society/issue) of cross-sector partnership. These four orders/loops correspond to the benchmark project efficiency (operational), project performance (tactical) and mission-related performance and issue-related performance. These orders/loops are as follows (see Figure 1):

• First order impact loop, which establishes the operational efficiency of partnerships through the effects of internal value added between inputs and throughputs/activities of the partnership. It links inputs with outputs.
• Second order impact loop, which establishes the tactical efficiency of partnerships through the effects of internal added value between the inputs and outputs, hence also capturing the operational level effects (first order impact loop). It links inputs with outcomes.
• Third order impact loop, which captures the value added of partnerships in accordance with its mission from inputs to outcomes and includes the interaction effects across the stages. It links mission with impacts.
• Fourth order impact loop that includes all the stages from input to impact. This allows the assessment of the overall value added obtained by the partnership and its contribution to the issue. However, these stages are the most complex to address because of the large number of levels of analysis and of interaction effects. It links the issue with impacts.

4. The 2008 Zaragoza International Exhibition and the Citizen Initiatives Pavilion

The growth model of the economies of most countries was producing an environmental deterioration on a global scale with a use of natural resources not sustainable. Social awareness of the States and citizens, and the acquisition of collective commitments aimed at modifying production methods and consumption habits became necessary. One way to try to increase this awareness was the organization of an international exhibition to serve as a meeting point between representatives of different countries with different cultures and ways of life in which they could propose strategies and interventions to achieve sustainable development.

The city of Zaragoza is the fifth biggest city of Spain with an estimated population of 666,880 inhabitants. It is the capital of Aragón, a medium-sized 47,669 km² region in the northeastern part of Spain with a GDP of 27,403 € per capita [33]. Zaragoza and Aragón are regularly selected in socioeconomic studies as representative Spanish city and region, respectively, as they have sectoral distribution and socioeconomic indicators similar to the mean of Spain [34]. On December 16, 2004, Zaragoza was elected by the BIE to host the EXPO 2008 ahead of two candidates: The Italian city of Trieste and the Greek city of Thessaloniki. Water management and sustainable development were issues that aroused (and continue to arouse) growing concern around the world. This thematic axis determined all the details of the EXPO: Its location in the meander of the river Ebro, its mascot chosen by popular vote (a drop of water called Fluvi acronym of the Latin “flumen vitae” or river of life), the shapes of the buildings, and the contents of the debates of experts during the months of the exhibition that gave rise to the Water Charter of Zaragoza. To channel and visualize social support for the EXPO, volunteering was promoted in Zaragoza. There were more than 40,000 registered volunteers who collaborated in the organization for up to 6 hours a day.

Finally, EXPO was held in Zaragoza from June 14 to September 14, 2008 about the global issue of water management and development models and whose theme was “Water and sustainable development”. It was an example of cross-sector collaboration with a double mission (or two objectives). First, to increase public awareness of problems associated with water and sustainable development. Second, to change the development model of Zaragoza to a more sustainable one, making the city more internationally known and modernizing its infrastructures.

The EXPO was regulated by the Bureau International des Expositions (BIE) and constituted a space dedicated to leisure, culture, reflection and exchange of knowledge about these problems among the different participants (106 countries, 346 NGOs, 62 public and private collaborating entities, 6 sponsoring partners and three international organizations). The event attracted more than 5.6 million visitors with an average of 60,763 visits per day [35].

The city of Zaragoza faced the enormous challenge of the construction of the EXPO site and the accompanying infrastructure with the collaboration of the public administration (Government of Spain, Government of Aragon and Zaragoza City Council), private entities and the third sector, which constituted the partnership Expo Social Zaragoza 2008. The site occupied 25 hectares of the meander of Ranillas next to the river Ebro, where the pavilions were built to locate the exhibition. Some of these buildings led to milestones in Spanish and world engineering. The infrastructure plan of the EXPO entailed investments of 2500 million euros with a whole series of road, railway, tourist and green interventions, resulting in a qualitative leap of modernity in the city [35].

The main economic data directly related to the event were the following. The investments for the site in land, housing development, access and others amounted to a total of 625 million euros. The income obtained from the sale of tickets, sponsorship and others was 236 million euros and the expenses in construction, communication, content, shows and others amounted to 291 million euros. It is estimated that the city of Zaragoza received some 900,000 tourists in relation to this exhibition [35].

Among the pavilions of the exhibition, the citizen initiatives pavilion (named FARO) was managed directly by NGOs. This is highlighted in its alignment with the theme of the exhibition “Water and sustainable development”, through its bioclimatic construction, and in its program of activities. FARO was the first chance non-governmental organizations have had to take responsibility for the contents and design of a Pavilion in an International Exhibition. It represented 346 civil organizations (developmental, ecological, social and cultural) and 62 public and private collaborating entities, on a local and Spanish level, as well as from the five continents. The civil organizations in Zaragoza were responsible for managing FARO with an aim and a project in common: an effort in participation and coordination which would change the world. FARO denounced the situation of millions of people who have no access to safe water or sanitation; it showed alternative solutions to the great challenge of sustainable coexistence between humans and nature; and it stimulated a desire to change to a sustainable way of life, spreading the spirit of the participating organizations and initiatives to the visitors.

5. The Partnership and Social Effects Monitoring and Evaluation

In this section, we elaborate a statistical tool to monitor and evaluate the cross-sector partnerships and their social effects from the opinions expressed by their intended beneficiaries. This tool is based on the methodology proposed by Van Tulder et al. [15] and it is applied to evaluate the short-, medium- and long-term effects of the EXPO. To that aim, we use structural equation models because of their ability to impute relationships between unobserved constructs (latent variables) from observable variables. In our case, we build a structural equation model based on the perceptions of the final beneficiaries of the EXPO: The citizens of Zaragoza. The measurement part of the model relates the items of the survey (observable variables) with the inputs, outputs, outcomes and impacts (latent variables) of the EXPO. The structural part of model is based on the diagram of Figure 1 and determines the relationships (direct and indirect) between the above effects by means of linear regression models which we use to estimate the operational, tactical and strategic efficiencies of the EXPO (see Appendix A for more details).

The input-activities are related to the pavilions and their contents, the volunteers who worked in them and aspects related to the organization of the EXPO. The outputs, outcomes and impacts are related to effects of the event in the short (during the event), medium and long term (both after the event), respectively. Besides, we calculate the operational and tactical efficiencies and the effectiveness associated with the two EXPO missions, evaluating the direct and indirect influences of the inputs on the outputs, outcomes and impacts indicators, obtained from the structural equation model.

We start by describing our sources of information and the variables used in the study. This is followed by a description of the methodology used and the results obtained.

5.1. Data

In collaboration with the EXPO, and within the citizen initiatives of FARO, the creation of a group was promoted to study the effects of the international exhibition in the city of Zaragoza of 2008. The group comprised voluntary experts on the subject and representatives of public and private institutions, NGOs and business organizations. The information to monitor and evaluate the partnership and its social impact on the city of Zaragoza was obtained by means of a survey of citizens who benefited from the project. The survey focused on a random sample of citizens of Zaragoza and was conducted in 2009, the year after the EXPO. This time lag was considered appropriate so that respondents could assess the effects of the event, as well as being able to remember as many aspects of it as possible.

The design of the survey took into account the two objectives of the EXPO: 1) to increase awareness of problems associated with water and sustainable development and 2) to change the development model of Zaragoza to a more sustainable one, making the city more internationally known and modernizing its infrastructures.

Table 1. Items analyzed together with the indicator to which they are associated ⁺.

Indicator	Item	Meaning
Input (Valuation of the physical elements and organization of the EXPO)	p.17_1	The architecture of the buildings was the most appropriate for the content of the exhibition
	p.17_2	The contents of the pavilions were appropriate for the theme of the international exhibition “Water and sustainable development”
	p.17_4	The EXPO volunteer staff appropriately transmitted the message of the EXPO on water and sustainable development to the visitor
	p.17_6	The organization and operation of the EXPO was adequate
Output EXPO Inner (Valuation of inner EXPO effects exerted on its visitors and as a consequence of its activities)	p.08_4	The EXPO spread knowledge of other countries and cultures about their relations with water and sustainable development
	p.08_5	The event was a place of leisure and recreation for citizens
	p.08_7	The managing entities demonstrated a great capacity of management giving an adequate image of the city
Output EXPO Outer (Valuation of outer EXPO effects on the daily life of citizens of Zaragoza)	p.09_1	The economic cost was compensated by the social benefits of the event
	p.09_3	There was a generalized rise in prices of basic goods during the EXPO celebration period due exclusively to the event
	p.09_4	There were serious problems in the city including daily traffic jams, increased crime, etc.
Output FARO (Valuation of effects of the FARO pavilion)	p.17_3	The FARO pavilion highlighted awareness among visitors of the problems related to water and sustainable development
	p.17_5	The message of the FARO pavilion positively influenced my awareness of the problems of water and sustainable development
Outcome NGOs (Valuation of short and medium-term effects of the EXPO and FARO on the work of NGOs)	p.17_7	My perception of the work of NGOs in the world was reinforced thanks to the EXPO
	p.17_8	The NGOs have shown that they can work together to solve these types of problems
Outcome City (Valuation of short and medium-term effects of the EXPO on Zaragoza)	p.08_1	Thanks to the EXPO, Zaragoza is now more internationally known
	p.08_6	Thanks to the EXPO, Zaragoza has achieved a modernization of its infrastructures that could not have been achieved without it
Impact Awareness (Valuation of long-term effects of the EXPO on awareness of water and sustainable development)	p.08_2	The EXPO created greater awareness among citizens of the problems related to water and sustainable development
	p.08_3	The EXPO created greater awareness among organizations (public and private) of the problems related to water and sustainable development
Impact City (Valuation of long-term effects of the EXPO on Zaragoza)	p.08_8	The citizens of Zaragoza believe that the future of the city has improved after the EXPO
	p.08_9	The citizens of Zaragoza feel more confident in their own possibilities after the EXPO

Note: ⁺ The answers to the items were measured on a 7-point Likert scale (1: strongly disagree, 2: disagree, 3: slightly disagree, 4: neutral, 5: slightly disagree, 6: agree, 7 strongly agree).

shows the selected items of the questionnaire, whose answers were codified on a 7-point Likert scale (a psychometric and symmetric scale which captures the level of agreement or disagreement for a series of statements from 1: strongly disagree to 7: strongly agree) together with the indicator to which they are associated and following the evaluation framework shown in Figure 1. The valuation of the inputs of the EXPO was carried out through an indicator obtained from the items p.17_1, p.17_2, p.17_4 and p.17_6 of the survey, which are related to the pavilions and their contents, the volunteers who worked in them and aspects related to the organization of the EXPO. Valuation of outputs was carried out through three indicators: The Output EXPO Inner indicator (built from items p.08_4, p.08_5 and p.08_7) which evaluates the effects exerted by the EXPO on its visitors as a consequence of its activities; the Output EXPO Outer indicator (built from items p.09_1, p.09_3 and p.09_4) which evaluates the effects exerted by the EXPO on some aspects related to the daily life of citizens of Zaragoza such as the economic cost of the event, price increases and the day-to-day operations of the city; and the Output FARO indicator (built from the items p.17_3 and p.17_5) which evaluates the effects of the FARO pavilion on the awareness of problems associated with water and sustainable development. Valuation of outcomes was made through the indicators Outcome City (built from p.17_7 and p.17_8 items) and Outcome NGOs (built from p.08_1 and p.08_6). Outcome City evaluates the short and medium-term effects of EXPO in the internationalization and modernization of Zaragoza, while Outcome NGOs evaluates the effects of EXPO and FARO on the recognition of the work carried out by NGOs to raise awareness among citizens about water problems and sustainable development. Valuation of the long-term impacts of these aspects was collected in the Impact City (built from items p.08_8 and p.08_9) and Impact Awareness (built from p.08_2 and p.08_3) indicators.

Table 2. Descriptive study of the items.

Indicator	Item	Obs.	Miss.	Min.	Q1	Median	Mean	Q3	Max.	St. Dev.
Input	p.17_1	448	7	2	5	5	5.30	6	7	1.15
	p.17_2	442	13	1	5	6	5.79	7	7	1.15
	p.17_4	443	12	2	4	5	5.20	6	7	1.27
	p.17_6	444	11	2	4	5	5.23	6	7	1.24
Output EXPO Inner	p.08_4	453	2	1	4	6	5.42	7	7	1.50
	p.08_5	453	2	2	5	6	6.06	7	7	1.14
	p.08_7	446	9	1	4	5	4.71	6	7	1.27
Output EXPO Outer	p.09_1	450	5	1	4	5	4.79	6	7	1.64
	p.09_3	448	7	1	2	3	3.19	4	7	1.70
	p.09_4	450	5	1	2	2	2.75	4	7	1.56
Output FARO	p.17_3	439	16	1	4	5	4.56	6	7	1.33
	p.17_5	440	15	1	3	5	4.52	6	7	1.46
Outcome NGOs	p.17_7	438	17	1	4	5	4.73	6	7	1.41
	p.17_8	433	22	1	4	5	5.17	6	7	1.30
Outcome City	p.08_1	453	2	1	4	6	5.31	6	7	1.34
	p.08_6	454	1	2	5	6	5.91	7	7	1.15
Impact Awareness	p.08_2	451	4	1	4	5	4.50	5	7	1.33
	p.08_3	445	10	1	3	4	4.21	5	7	1.36
Impact City	p.08_8	449	6	1	4	5	5.17	6	7	1.29
	p.08_9	445	10	1	4	5	4.95	6	7	1.42

shows the results of a descriptive study of the answers to the questionnaire. The number of surveyed citizens was 455 of which 396 (87%) answered all the items, the percentages of non-response to each item oscillating around 2%. The most frequent profile of the respondents corresponds to a woman (52.3%), between 26 and 55 years of age (54.5%), with university studies (42.6%) and salaried (44.2%) (see Supplementary Material for more details). In addition, most of the respondents visited the EXPO (83.3%) and only 16.7% of them were volunteers.

The scores obtained in the formulated items were, in general, medium-high with average scores between 4.2 and 5.9 (Table 2). Only items p.09_3 and p.09_4 obtain scores below these levels (3.19 and 2.75, respectively); nevertheless, it should be taken into account that these items were drafted negatively.

The highest valuations corresponded to items related to the short and medium-term effects on the city (Outcomes City), the inner results of the EXPO, i.e., the effects exerted on its visitors and as a consequence of its activities (Output EXPO Inner) and the inputs, with mean scores larger than 5.3 for most of these items. On the contrary, the lowest valuations corresponded to items related to the effects of FARO (Output FARO) and to the long-term EXPO effects on the awareness of society of the problems related to water and sustainable development (Impacts Awareness) with most of items related to these indicators having mean values lower than 4.6 points.

5.2. Structural Equation Model

Figure 2 shows the perceptual diagram of the structural model used to measure the short-, medium- and long-term effects of the EXPO. The model is based on Van Tulder et al. [15] where a very general framework to carry out an assessment of cross-sector partnership activities performed (inputs) and the short (outputs), medium (outcomes) and long (impacts) term effects are described.

The equations of the model are given in Appendix A. The method used for the estimation of their coefficients has been the maximum likelihood using the R statistical package lavaan [36].

Table 3. Measurement model.

Indicator	Item	Factor loadings (aij)			Intercepts (μi)			Error Variances $\mathbf{\left(}{\mathsf{\sigma }}_{\mathbf{i}}^{\mathbf{2}}\mathbf{\right)}$
		Estimate	SE	p-Value	Estimate	SE	p-Value	Estimate	SE	p-Value
Input	p.17_1	0.764	0.052	0.000	5.297	0.054	0.000	0.743	0.059	0.000
	p.17_2	0.950	0.056	0.000	5.187	0.060	0.000	0.713	0.065	0.000
	p.17_4	0.735	0.053	0.000	5.779	0.054	0.000	0.779	0.061	0.000
	p.17_6	0.812	0.056	0.000	5.234	0.058	0.000	0.861	0.068	0.000
Output EXPO Inner	p.08_4	0.872	0.054	0.000	5.419	0.070	0.000	0.861	0.082	0.000
	p.08_5	0.608	0.042	0.000	6.055	0.054	0.000	0.640	0.053	0.000
	p.08_7	0.534	0.050	0.000	4.714	0.060	0.000	1.106	0.083	0.000
Output EXPO Outer	p.09_1	1.048	0.067	0.000	3.209	0.077	0.000	1.283	0.113	0.000
	p.09_3	-1.204	0.070	0.000	3.205	0.080	0.000	1.043	0.104	0.000
	p.09_4	-1.156	0.061	0.000	2.746	0.073	0.000	0.716	0.088	0.000
Output FARO	p.17_3	0.553	0.049	0.000	4.550	0.063	0.000	0.956	0.078	0.000
	p.17_5	0.693	0.059	0.000	4.504	0.069	0.000	0.876	0.084	0.000
Outcome NGOs	p.17_7	0.470	0.071	0.000	4.715	0.067	0.000	0.557	0.064	0.000
	p.17_8	0.444	0.066	0.000	5.138	0.062	0.000	0.433	0.054	0.000
Outcome City	p.08_1	0.374	0.068	0.000	5.313	0.063	0.000	0.814	0.071	0.000
	p.08_6	0.203	0.041	0.000	5.908	0.054	0.000	1.034	0.073	0.000
Impact Awareness	p.08_2	0.713	0.043	0.000	4.503	0.062	0.000	0.437	0.057	0.000
	p.08_3	0.746	0.047	0.000	4.217	0.064	0.000	0.397	0.060	0.000
Impact City	p.08_8	0.518	0.045	0.000	5.167	0.061	0.000	0.541	0.055	0.000
	p.08_9	0.618	0.057	0.000	4.936	0.067	0.000	0.412	0.065	0.000

contains the estimations of the intercepts, factor loadings and variances of the error terms of the measurement model while

Table 4. Structural model.

	Output EXPO Inner				Outcome City				Impact City
	Estimate	SE	p-value		Estimate	SE	p-value		Estimate	SE	p-value
βOutput_Inner,Input	0.892	0.091	0.000	βOutput_Inner,Outcomes_City	1.827	0.379	0.000	βOutcomes_City,Impacts_City	0.596	0.125	0.000
								βOutcomes_NGOs,Impacts_City	0.154	0.046	0.001
	Output EXPO Outer				Outcome NGOs				Impact Awareness
	Estimate	SE	p-value		Estimate	SE	p-value		Estimate	SE	p-value
βOutput_Outer,Input	0.521	0.068	0.000	βOutput_Faro,Outcomes_NGOs	1.445	0.262	0.000	βOutcomes_City,Impacts_Awar	0.382	0.079	0.000
	Output FARO							βOutcomes_NGOs,Impacts_Awar	0.174	0.041	0.000
	Estimate	SE	p-value
βOutput_Faro,Input	1.280	0.138	0.000
σOutput_City,Output_Expo	0.475	0.056	0.000
RMSEA	0.085	95% CI (0.078, 0.091)
SRMR	0.065
CFI	0.889
TLI	0.868

shows the estimations of the coefficients of the structural model. The factor loadings of the measurement model are all significantly different from zero and with the expected sign (see Table 3). Besides, most of the coefficients of the structural model (see Table 4) are coherent with the direct relationship of the logic model of Figure 1, revealing the existence of significant direct influences of the Inputs on the Outputs, of the Outputs on the Outcomes and of the Outcomes on the Impacts for the two objectives of the EXPO.

The only exception is the absence of a significant direct relationship between the Output EXPO Outer and the Outcomes City. This is due to the significantly positive residual covariance (0.475) between the Output EXPO Inner and Output EXPO Outer, reflecting the influence of the common aspects related to the internal and external EXPO management, which highlights the existence of a significant indirect relationship between Output EXPO Outer and the Outcomes City through Output EXPO Inner.

It is worthwhile highlighting the significant direct influences of the Outcomes NGOs and Outcomes City on the long-term effects of the EXPO revealing the existence of positive synergies between its two main objectives: raising awareness of problems related to water and sustainable development and changing the development model of Zaragoza to a more sustainable one, making the city more internationally known and modernizing its infrastructures.

Finally,

Table 5. Effects of the EXPO in the short-, medium- and long-term.

Efficiency	Estimate	SE	p-Value
Operational EXPO (1)	0.892	0.091	0.000
Operational City (2)	0.521	0.068	0.000
Operational FARO (3)	1.280	0.138	0.000
Tactical City (4)	1.629	0.334	0.000
Tactical NGOs (5)	1.851	0.330	0.000
Strategic City (6)	1.256	0.147	0.000
Strategic Awareness (7)	0.945	0.092	0.000

(1) β_{Output_Inner,Input} (2) β_{Output_City,Input}. (3) β_{Output_Faro,Input}. (4) β_{Outcome_City,Output_Inner}* β_{Output_Expo,Input}. (5) β_{Outcome_NGOs,Output_Faro}* β_{Output_Faro,Input}. (6) β_{Impact_City,Outcome_City}* β_{Outcome_City,Output_Inner}* β_{Output_Inner,Input} + β_{Impact_City,Outcome_NGOs}* β_{Outcome_NGOs,Output_Faro}* β_{Output_Faro,Input}. (7) β_{Impact_Awareness,Outcome_City}* β_{Outcome_City,Output_Inner}* β_{Output_Inner,Input} + β_{Impact_Awareness,Outcome_NGOs}* β_{Outcome_NGOs,Output_Faro}* β_{Output_Faro,Input}.

shows some goodness-of-fit measures of the model. Concretely, we provide the Root Mean Square Error of Approximation (RMSEA) and the Standardized Root Mean Residual (SRMR) which compare the observed and the hypothesized covariance matrices of the model. Both are less than 0.1 which is the limit that separates an acceptable fit of a good fit. Besides we provide the Comparative Fit Index (CFI) and the Tucker-Lewis Index (TLI) which compare the fit of the model with the independence model. We notice that both indices have values larger than 0.95 that is considered the limit to have an acceptable fit.

Table 5 contains the estimations of the operational, tactical and strategic efficiencies of the EXPO which measure, respectively, its short-, medium- and long-term effects. These effects have been estimated using the mathematical expressions given at the bottom of this table (see also the Appendix A where it is shown how these expressions have been obtained). All the effects are significant and positive. The larger values correspond to the short- and medium-term effects on awareness about problems related to water and sustainable development and on the NGOs in FARO. In the long term, however, respondents consider that impacts on the city are greater.

5.3. Analysis of Impacts after EXPO

Today, 10 years after the celebration of the EXPO, the impacts of its two objectives can be clearly seen. First, there was an increase in public awareness of and commitment to the problems associated with water and sustainable development [35], both at the level of the city itself and outside it. Within the city, important intangibles were achieved that are still valid a decade later. Among other examples, we can highlight the effect that the EXPO had as a vehicle of unification and integration, the consolidation of the volunteer movement (which even today, is an active group that continues to collaborate with the city council in organizing all kinds of events), and the changes in urban mobility with the reduction of motorized traffic in the city center and the increasing use of the bicycle. As a consequence, Zaragoza now has a municipal bicycle rental service and the number of kilometers of bike lane has risen from 12 before the EXPO to the current 131. Outside the city, the impact of the EXPO on commitment to the problems associated with water and sustainable development has been considerable. Perhaps the most relevant, due to its international projection, was achieved by the Water Tribune, a forum of experts from all over the world, organized during the EXPO. The conclusions of the work carried out by these experts gave rise to the Charter of Zaragoza on Water, which subsequently led to the General Assembly of the United Nations officially forming the Water Council and recognizing people’s rights to water and sanitation in July 2010. These rights were reflected in the Sustainable Development Goals (SDGs) whose number 6 is “Guaranteeing the availability of water and its sustainable management and sanitation for all”, without forgetting that the intellectual legacy was collected and systematized in the Blue Box, which makes accessible thousands of documents -texts, presentations, videos, photographs, news, and other works- that emerged from conferences, workshops, debates and various performances held in the Water Tribune or derived from its various instruments.

The EXPO also achieved a high impact on its other major objective, changing the development model of Zaragoza to a more sustainable one, making the city more internationally known and modernizing its infrastructures. These impacts were indicated by Serrano et al. [37], Barlés and Anso [38] and Duarte et al. [39], who examined the associated investments and tourism expenditures, finding that the effects of the EXPO were positive. These effects took place, basically, in the services sector [40] and although the productive structure was not modified, it changed the intensity of the relationship between sectors [39]. The EXPO made it possible to carry out projects that had been pending for decades and their role in the transformation of Zaragoza was transcendental [41,42]. The city obtained the public investment needed to build ring roads, a terminal at the airport, bridges and footbridges on the three rivers that bathe the city (Ebro, Gállego and Huerva), and to recover their banks. Furthermore, the EXPO site itself provides a physical space that today constitutes the new lungs of the city with 125 hectares of green area. This space is a new zone of familiar leisure in a city that, until then, lived with its back to the Ebro and contains half a hundred kilometers of green walks along the urban sections of the three rivers. Moreover, the reconversion of the EXPO enclosure into a business park, which was planned for when the pavilions and buildings of the exhibition were designed, has allowed its rapid transformation. These buildings are now an administrative center of the city giving work to more than 3000 people. The EXPO has also caused an impact outside the city. Currently, Zaragoza is much better known at all levels although causes that are alien to the exhibition itself and difficult to determine at present may have contributed to this knowledge. According to the National Institute of Statistics, the number of national visitors to the city has increased by 3.7%, that is, from 52,569 visitors a month in the years prior to the international exhibition to 54,541 in the following years. Foreigner visitors have increased by 53.5% from 12,940 to 19,875 visitors per month, respectively [33].

Summarizing, the EXPO 2008 was an event that favored substantial growth, both economic and social, through the change in urban growth and development models, thus promoting new urban projects and implementing good practices in environmental matters, fundamentally in water management and conservation and sustainable development [43]. In this way, Zaragoza is now a new city, more integrated, more complex, with greater exchange capacity, more sustainable, trained for new challenges, with greater capacities, intercultural and open [44].

6. Conclusions

This study proposes an empirical procedure for measuring, through the perceptions of their final beneficiaries, the short-, medium- and long- term effects of cross-sector partnerships and their implications for efficiency and effectiveness. The procedure uses a structural equation model based on the methodology proposed by Ebrahim and Rangan [14] and Van Tulder et al. [15] and, to our level of knowledge, is the first time that it has been used in this context in the literature. The methodology is applied to the assessment of the impacts of the 2008 Zaragoza International Exhibition “Water and sustainable development” on its two objectives: to increase public awareness of and commitment to the problems associated with water and sustainable development, and to change the development model of Zaragoza to a more sustainable one, making the city more internationally known and modernizing its infrastructures.

To do so, we have used the opinions of a sample of the beneficiaries of the cross-sector partnership in a specifically designed survey. Our results show that, in the opinion of the EXPO beneficiaries, there were positive long-term synergies between the achievements of its two objectives. All the estimations of the operational, tactical and strategic efficiencies of the EXPO that measure the effects of the inputs of the EXPO in the short, medium and long term were significantly positive. The greatest short- and medium-term effects occurred in the awareness of the problems associated with water and sustainable development through the NGOs in the FARO pavilion. However, in the long term, the beneficiaries of the exhibition considered that the impacts on the reaching of the above objectives would be somewhat smaller. Thus, we can conclude that, according to the opinion expressed in 2009 by the beneficiaries of the EXPO, the event managed to fulfill its two main goals in the short term and, to a lesser extent, that it would achieve its objectives in the medium and long term.

However, despite the success achieved and the fulfillment of its double mission, today there is a sense that there are still many things to do, especially at an ideological and cultural level. The concern of the institutions for the use of the EXPO site and its adaptation for productive uses was a priority instead of promoting the intellectual legacy of the EXPO, in a period of crisis that lasted several years. This corroborates the perceptions of our respondents about the impacts of the EXPO, which were larger on the city than on citizen awareness. All these facts are in tune with the perceptions provided by the beneficiaries of the EXPO in 2009, thus giving indirect validity to our study and the proposed methodology.

In view of these results, and in order to promote the EXPO intellectual legacy and make Zaragoza a point of reference at a global, national and local level, it might be convenient to establish training and education centers on the proper use of natural resources to change the habits of citizens and, especially, of future generations. We know that the post-EXPO association is trying to promote and keep this legacy alive by presenting different lines of action to politicians.

Nevertheless, the present study can be very useful to politicians and leaders. In addition to the massive investments that EXPO brought to Zaragoza, a very important aspect to assess the success of an event like this is what contributes to the leisure economy of the region and to the creation of a new image of the territory. In this sense, knowing the assessment of citizens regarding the event is fundamental and one of the factors that political leaders should take into account. Another aspect that explains the success of a great event is its contribution to social cohesion and progress of society. This is achieved by giving a satisfactory response to some serious current problem (for example, the lack of sustainability) and establishing a new scale of values. The results analyzed reveal that in all previous aspects the EXPO achieved its objectives even though it is necessary to continue with the awareness of the new generations.

Even though the representativeness of the specific case analyzed in the paper is limited and the proposed methodology needs to be validated in other contexts, we believe that, being based on a framework as general as that proposed in Van Tulder et al. [15], our procedure could be used as a useful starting point to establish a measurement method of the effects, efficiency and effectiveness of partnerships through the opinions of their final beneficiaries. Furthermore, the methodology is flexible enough to include other aspects that have not been considered here, such as the perception of other stakeholders, changes in the goals of the players over time or the experience partners have in the partnership, by including items aimed at measuring these aspects in additional surveys. These possible extensions are contemplated as futures lines of research.