Introduction

Air pollution is a serious issue at the world level that needs to be addressed (Molina & Gurjar, 2010). World Health Organization (2021a) reports that air pollution in major cities plus the poor indoor air quality in people’s homes can lead to cancer, respiratory and heart diseases. While traffic and heavy manufacturing are major causes of air pollution in cities, indoor air quality is influenced by poor ventilation, indoor sources, among other factors. World Health Organization (2021b) also reports that 2.6 billion people of low and middle-income countries around the world are exposed to indoor air pollution. Several studies showed that air pollution with high concentrations of polycyclic aromatic hydrocarbons, particulate matter (PM) with diameter less than 2.5 \(\upmu\)m (PM2.5) and NO2 have been affecting children and adults’ nervous systems, showing an increase in attention deficit hyperactivity disorder (ADHD), autism and lower intelligence quotient levels in children, as well as dementia, depression and episodic memory in adults (Sram et al., 2017). Moreover, it is also recognized that higher CO2 concentrations in classrooms indicates poorer ventilation and higher levels of indoor air pollutants which impacts students’ performance (Coley et al., 2007; Wargocki et al., 2020). Due to poor ventilation in schools, most children in Lisbon are exposed to high concentrations of PM2.5 and PM10 (Faria et al., 2020). Air pollution can also impact the human immune system and make it more susceptible to other diseases (Yang et al., 2020). Aerosols can also contribute to climate change (Masalaite et al., 2020).

In terms of indoor air quality perception, it has been shown that families with low incomes perceive the air quality through mostly, vision, smell, and comfort in terms of room temperature (Kim et al., 2019). They associate a good indoor air quality with how tidy and clean the rooms are and they also think that products such as deodorants, scented candles and cleaning products are ways to improve indoor air quality. This shows the lack of awareness these low-income families have, given the fact that human senses, like vision and smell, cannot detect every pollutant. Moreover, scented candles are harmful to our health due to the particles they release when lit (Massoudi & Hamidi, 2017). In recent years there has been an increase in the number of studies evaluating people’s risk perception of air pollution. Several of these studies associate a better risk perception of air pollution from people who have had bad experiences previously with pollution. Overall, studies appeal to the implementation of policies with the purpose of reducing the impacts caused by air pollution, but such legislation will only be adopted if populations around the world show their willingness to change their behaviours towards a more pro-environmentalist behaviour (Cori et al., 2020). These results show that efforts are needed to raise awareness of people regarding air pollution and encourage them to practice more pro-environmental behaviours.

One way to teach new concepts is through serious games or gamification (Abt, 1987). Researchers have shown this methodology is capable of modifying consumers’ behaviour towards more environmental friendly ones (Morganti et al., 2017).

This paper presents Problems in the Air, a serious game aimed at promoting pro-environmental behaviours related to air pollution in children. In this game, the child is asked to control a character that needs to deploy a set of sensors across several areas of an imaginary city with the goal of monitoring air quality. Through the character, the child interacts with several non-playable characters representing a scientist and friends, providing information and dilemmas to be solved by the child. One strength of the game is its ability of being customized by the child’s educator (e.g., a teacher). Customization includes the dialogues between the main and secondary characters.

To evaluate the ability of the developed serious game to increase children’s perception about air pollution, an identical pre-test and post-test questionnaire about pollution was elaborated and filled by the children before and after playing the game, respectively. This allowed us to assess whether the children would have a better perception after playing the game or not. To evaluate the game’s usability, a game experience questionnaire to evaluate several game aspects was filled by the children after playing the game. Usability scores were obtained using the System Usability Scale (SUS) score technique from the Likert items’ questionnaire results. Evaluation was done with twenty elementary students between seven and nine years old. Comparing pre- and post-test questionnaires results about air pollution awareness show that the game provides positive effects towards improving children’s perception of air pollution. Regarding the game’s usability, an acceptable mean usability score was obtained.

Related work

There are air pollution tools that provide information at a world level. Examples include the World Air Quality IndexFootnote 1. and BreezometerFootnote 2. websites. They use information from ground static stations equipped with costly but highly sensitive sensors. Due to the reduced number of these measuring stations, further processing is required in order to obtain city block resolution maps.

The advent of low cost sensors have sparked a series of projects to obtain higher resolution pollution maps (Santana et al., 2020) (Hasenfratz et al., 2015). In these cases, sensors are deployed in public transportation vehicles. Given the high volume of generated data, researchers have applied machine learning to make predictions (Mariano et al., 2021).

These tools can be used, for example, to advise citizens on when or where they should avoid outdoor activities.

Concerning air pollution visualisation, the work described in Roldán-Gómez et al. (2020) used the Unity Engine to create a 3D representation of air quality data collected by a group of drones. They used virtual reality to present air pollution in a more fun and interactive way. Additionally, the data presented includes traffic, pedestrian density, and temperature. Another example of a 3D-based air quality visualisation tool is CityOnStats (Teles et al., 2020). This tool provides the user with intuitive access to air quality data collected by a mobile sensor network and represented with a diverse set of graphical representations. In CityOnStats, the user controls a city bus to navigate around a virtual representation of the monitored area, generated from the urban topology available in Open Street Map databasesFootnote 3.

Our work is also related to previous work on the serious games and gamification of applications. The term serious games is known to be mentioned for the first time around the 70’s in Abt (1987), where the author suggests that games should not only be used as entertainment but also to improve education or a specific skill. In the 20th century, dozens of serious games were developed and released for different markets (Wilkinson, 2016). At the turn of the century, from 2002 to 2010, there were already more serious games released than between 1980 and 2001 (Djaouti et al., 2011). At an academic level, in recent years, there also has been a gradual increase in the number of publications about serious games (Zhonggen, 2019). Concretely, academic research has shown that the use of serious games and gamification of applications is a way of leading consumers to practice pro-environmental behaviours, such as energy efficiency and consumption awareness (Morganti et al., 2017). However, some of these behaviours ended up being short term, which is not ideal.

Examples of gamification of applications include Prophet et al. (2018), in which air quality is presented using as representation the shape of a tree. Users interact with the application through augmented reality. They have to take care of the tree, which grows depending on local air quality. In Polys et al. (2017) the authors present an example of a Geographic Information System (GIS) that displays local information that has been gamified in order to increase environmental awareness. In Briones et al. (2018) the authors proposed a gamified application to encourage citizens to recycle more and get rewards in exchange.

In elementary schools, surveys evaluating teachers and students’ opinion about the use of educational games, have shown that both groups agree that games could be used as a motivational tool for learning (Andic et al., 2018). Teachers pointed out the importance of these games being customisable so that they can be tailored for lessons. There have been many attempts to raise awareness about air pollution and climate change through serious games. For instance, in Feldpausch-Parker et al. (2013) the authors created an educational game to raise students’ awareness about the impact of CO2 emissions. Other studies managed to improve awareness and knowledge about health hazards caused by air pollution. For example, in Carducci et al. (2016) the authors managed to increase knowledge and awareness of these health hazards to elementary students using leaflets, cartoons, and video games. Serious games have also been developed to teach middle school students about the negative effects of exposure to toxic chemicals (Klisch et al., 2012). Games and psychological distance have also been used together to shift people’s behaviour to a more pro-environmental stance (Fox et al., 2020). Another example of using games is (Moore & Yang, 2020) where authors check if people, that watch a demonstration of an ecology game or that actually play the game, are more prone to exhibit pro-environmental behaviour. It is also possible to evaluate the children’s perception of risk regarding climate change (Puttick & Tucker-Raymond, 2018).

Evaluation of children’s perception about pollution can also be studied with activities like drawing. For example, in Shepardson (2006) the author asked students of several grades to give their own idea about what an environment is through drawings. The author examined what components of the environment (processes, systems, entities) children were aware of. Most students ended up drawing scenes containing animal life in it. Using the same technique, the work presented in Özsoy and Ahi (2014) evaluated elementary students’ perception of what the environment will be in a near future. It was found that students would expect a clean, a polluted, or a technological environment depending on where they live. This drawing technique was also applied to preschool children (Duran, 2021), revealing that children manage to better express pollution perception with drawings, when compared to with interviews.

When creating a game, different approaches can used. One of them is participatory design (PD) (Wanick & Bitelo, 2020) This approach does not prescribe a rigid sequence of steps that must be followed. Rather, the focus is on the establishment of common knowledge between all the stakeholders so that all people involved understand each other. PD has been used in the context of game development with special emphasis on game players. When designing a game with the goal of teaching new concepts to children, Dodero et al. (2014) advocates for a co-design approach, where both teachers and children actively participate in the design process.

Critical analysis

This section shows that there is considerable previous work on monitoring and prediction of air quality in urban environments, as well as on visualization of sensor-based air pollution data in 3D environments, which facilitates users’ understanding and perception of air pollution. Serious games and gamification of applications have also been developed with the purpose of fostering user engagement while learning important air quality related issues. However, none of these previous serious games focused on teaching the engineering aspects of air quality monitoring, that is, on the role and characteristics of air sampling and sensing. Moreover, these previous serious games do not offer teachers with customisation capabilities, which has been identified as an important feature (Andic et al., 2018). Our serious game Problems in the Air, presented in this paper, fills these gaps in the literature. Concretely, the game allows children to learn the engineering aspects of air quality monitoring with contents customised by their teachers. This way, the game promotes air pollution sensing and increases children’s perception regarding air pollution.

The game: problems in the air

Game design

A PD approach was followed during game development. Concretely, an environmental expert on air quality monitoring was included in the iterative design loop. The conclusions of early brainstorming meetings were that the game should be about a character whose goal is to monitor air quality in a small city and its surroundings.

The motivation to choose this design approach was to guarantee the scientific correctness of the game’s content. That is to say, all terms, scenarios, dilemmas are scientifically sound. This aims to ensure that children know the correct terms used in the environmental field. While coa -design methodology also includes the target audience of the game, in our case, the environmental expert has experience working with children and in giving presentations to high school students. As such, our expert choice indirectly brought children’s viewpoint.

Storyline, characters and city

The storyline devised during game design is as follows. The main character lives in a city. One of his friend is a scientist who one day realises that people are getting sick from air pollution related problems. The main character and the scientist decided to create a group named Friends for the Environment in order to study air pollution.

Four friends of the main character also joined the environmental group. Due to their lack of knowledge about air pollution, they are going to be faced with dilemmas that may affect their health. The main character is going to help his friends solving the dilemmas by telling what they should do in each case.

The scientist knows what may cause the air pollution problems. He is going to task the main character to go around the city in his bike and place air quality sensors. Every time a new sensor is placed, the scientist explains how a specific city area or location can contribute to air pollution and to health issues.

Since one of the main gameplay goals of the game is to monitor the air quality around a city, several common city areas, such as downtown, residential zones with houses and flats, industrial zones, forest zones and parks were created. Each area has an air pollution level consistent to what is expected in real life, namely: (1) an industrial area with the worst pollution levels; (2) a downtown area, located in the center of the map, with above average pollution levels; (3) a residential area with flats with an acceptable pollution level; (4) a residential area with house and gardens with a low pollution level; and (5) a forest area also with a low pollution level.

When the game starts, the player is presented with the welcome screen (see Fig. 1). The top area displays the four player’s friends (top row), the scientist (bottom row and the player’s character. The city can be seen in the centre, while on the left side there is a menu.

Fig. 1
figure 1

Welcome screen showing game characters and game map

Full size image

Game goal

Problems in the Air is a serious and educational game. The game goals are to promote pro-environmental behaviours in children and to increase environmental awareness. These goals are achieved with two mini-games:

  • sensor mini-game: where the main character places sensors in different city locations and learns about typical air quality levels;

  • dilemma mini-game: where the main character is faced with dilemmas that have environmental consequences or with questions about the environment.

The player will only be able to solve a dilemma or question if he has earned enough skill points so far. To earn these skill points, the player has to place sensors in the city by solving instances of the sensor mini-game. Hence, both mini-games complement each other in terms of learning goals. While the dilemma mini-game educates by having the player selecting correct answers, the sensor mini-game educates via exploration, as in it the player needs to place sensors and discover why a given city area exhibits a given pollution level.

Figure 2 shows a screenshot of the main window of the game. The top bar shows the progress in both mini-games: skill points are received when a sensor is placed, and score points are obtained when a dilemma is answered. The central part shows the city map with the zones that are represented. A video gameplay of the game can be seen in YouTubeFootnote 4.

Fig. 2
figure 2

Main screen with city map, player’s friends, and game controls

Full size image

Sensor minigame

The main goal of the sensor mini-game is to teach the player how air quality correlates with the different characteristics of each city zone. To achieve this, the player is asked to ride a bicycle and install several sensors across the city zones. Whenever a sensor is installed, the scientist presents the sensors’ readouts to the player and explains what is most likely causing them. The scientist also informs the player about the zone’s safety in terms of air quality. The player is allowed to install three types of sensors.

The first type represents fixed monitoring stations, which once installed cannot be picked up again. This type of sensor is highly accurate but considerably expensive and large and, thus, the player can only install a few of them. In real-life, monitoring stations are the most common approach to monitor air quality in cities. The second type available to the player is portable and expensive. Although not as accurate as fixed monitoring stations, this type of sensor has the advantage of providing sufficiently accurate readings in real-time everywhere and, hence, highly useful in real-life monitoring applications. Given their lower cost, sensors of this type can be deployed in larger quantities by the player. Finally, the third type of sensor represents real-life low-cost sensors, which provide the lowest accuracy, such as the one being developed in the ExpoLIS project (Santana et al., 2020). Their lower cost render easy to replicate them and, thus, to scatter around the city. The graphical representations of the three types of sensors are depicted in Fig. 3.

Fig. 3
figure 3

Graphical representations of three types of sensors that the player can install across the city: a fixed monitoring station; b expensive portable sensor; c low-cost portable sensor

Full size image

Flowchart

In the sensor mini-game, the player is given a set of sensors and a set of locations to monitor. During the mini-game, the scientist will, from time to time, remember the player what is the next location to place a sensor. When all locations have been monitored, the scientist congratulates the player, and the mini-game ends.

Figure 4 shows a flowchart of this mini-game. There is a main loop that starts when the player picks a sensor. This trigger the creation of an internal object, SensorArea that represents the area to be monitored. When the player places the sensor, the scientist appears and gives an explanation of the sensor reading. The loop repeats if the sensor is mobile and there are more locations to be monitored or if there are more sensors to be placed. The locations to be monitored and the text with the explanations, that are shown, are located in a resource file.

Fig. 4
figure 4

Flowchart of sensor mini-game

Full size image

Pollution tile maps and sensor monitoring

In the sensor mini-game, the player is asked to monitor air pollution in the city. In the game, a simplified air pollution model is used. The city is represented by a rectangular grid, i.e., a layer in Unity’s terminology. Each grid cell can contain a road, a building, a garden or vegetation. Air pollution is also represented by a rectangular grid (see Fig. 5a and b). Each grid cell can contain one of five pollution values (see Table 1). These values were taken from the Air Quality Index (AQI) websiteFootnote 5.

Table 1  Colour-based air quality representation used in the game
Full size table
Fig. 5
figure 5

Air pollution grid maps: a outdoor; b indoor

Full size image

Whenever the game needs to present the pollution measured by a virtual sensor located in a grid cell g, it samples an area centered on g, a(g), using the simplified air pollution model. For each possible pollution level p, the number of grid cells in a(g) with pollution level p is counted, n(a(g), p). To that number, noise taken from a random Gaussian distribution \(\hat{n}\sim G(\mu ,\sigma )\) is added, \(n(a(g),p)+\hat{n}\), with its parameterization defined according to the expected sensor’s inaccuracy (i.e., \(\sigma\) is highest for the low-cost sensor and lowest for the fixed monitoring station). Finally, the pollution level of grid cell g is: \({\text {*}}{argmax}_{p} (n(a(g),p)+\hat{n})\). When g corresponds to a location outdoors, the area a(g) is defined as a \(5 \times 5\) grid. Conversely, when g corresponds to an indoor location, a \(3 \times 3\) grid is used. This difference reflects the different air pollution diffusion levels in both situations.

The sensors shown in Fig. 3b and c are located in an area with mostly pollution level “moderate”. While the portable expensive sensor does indeed read “moderate”, the cheap sensor sometimes reads “unhealthy for sensitive groups”. This fluctuation reflects the higher inaccuracy characterising the portable sensor.

Dilemma mini-game

The dilemma mini-game consists of educational questions and dilemmas about the good and bad practices regarding air pollution. Dilemmas and questions appear in the city map as markers that the player must head to. When the player reaches the marker, a text box is shown along with two possible answers, as illustrated in Fig. 6. A dilemma answer can impact the main character’s (controlled by the player) friend’s health in a positive or negative way. In game, this is illustrated with an animation showing either a happy or sad friend’s face, see Fig. 7a and b, respectively. A question answer (non-dilema) does not affect friends’ health.

Fig. 6
figure 6

Dilemma text box with its answers

Full size image
Fig. 7
figure 7

Player’s friend reaction of a dilemma answer. a positive reaction; b negative reaction

Full size image

A flowchart of the dilemma mini-game is shown in Fig. 8. Again there is a main loop that is repeated until all dilemmas have been presented to the player. The player is only allowed to see the dilemma if he has enough skill points. After answering the dilemma an appropriate animation is shown, and the loop iteration finishes. The dilemmas and questions text and their answers are stored in a resource file.

Fig. 8
figure 8

Flowchart of dilemma mini-game

Full size image

Mini-games content

To allow teachers to tailor the scientific contents of the mini-games to their students (e.g., dialogues, questions and dilemmas, monitoring goals, sensor readings explanations), these are stored in editable resource files, one per mini-game, that are read by the game without requiring recompilation. The default contents of these resources files shipped with the game were created and curated by environmental experts during participatory design sessions. Teachers can then edit the resource files to add new locations to be monitored by the player, to add new dilemmas in order to explain concepts related to air pollution not addressed by default, and adjust the language and explanations to better match their students’ characteristics (e.g., explanations regarding the causes for a given sensor readout). The resource files are stored in Extensible Markup Language (XML) format. The following presents a brief description of these files’ structure. For further details and examples please refer to Relvas (2021).

The XML format is a language suitable for machine interpretation. It has a tree like representation where a root element can have elements as children. These can also have other children (they are great-children of the root element). The structure of a XML file can be represented as an inverted tree (as can be seen in Fig. 9 with one root element and four branches and in Fig. 11 with one root element, one child element, and one great-child element). Elements are represented by a box. In the top of the box is the element name and bellow are the optional element attributes.

Figure 9 shows the structure of the resource file for sensor mini-game. An example of a XML file is shown in Fig. 10. The root element is SensorsInfoXML and marks the XML has containing information for our game. The root element has four types of child elements:

  • Dialogue elements containing the text that is displayed when the player places a sensor;

  • Location elements containing the city grid cell coordinates where sensors can be placed;

  • GuideDialogue elements containing two texts. They are shown when the player is taking too long to pickup a sensor or to place a sensor. These texts recall the player what should be done next;

  • SensorLocationDialogue element connecting the above three elements as it specifies which location should be monitored next, the text that is shown and explains the sensor reading, and the text that guides the player. Additionally, it contains the location name and whether the location is indoor or outdoor. This element contains the location name instead of element Location because the same city grid cell may house several points of interaction.

Fig. 9
figure 9

XML resource file format for the sensor mini-game

Full size image
Fig. 10
figure 10

An example of the XML resource file for the sensor mini-game

Full size image

Figure 11 shows the XML format of the resource file for the dilemma mini-game. Figure 12 shows an example of a XML file. The root element is DilemmaContainer and marks the XML file as containing information for the dilemma mini-game. The root element only contains one type of element that represents the existing dilemmas. The structure of this file is:

  • Dilemma elements containing the text of the dilemma or question, the friend that is mentioned in the text, and the required skill points;

  • Answer elements that are children of Dilemma elements. They represent one of the possible answers to a dilemma. An Answer element contains the label that is shown in the text box (illustrated in Fig. 6), and the pollution impact on the friend;

  • ChildScientistDialogue elements that are children of Answer elements. These children represent a sequence of texts that the scientist says if the player chooses that answer. This sequence serves to explain the consequences of picking the corresponding answer.

Fig. 11
figure 11

XML resource file format for the dilemma mini-game

Full size image
Fig. 12
figure 12

An example of the XML resource file for the dilemma mini-game

Full size image

Evaluation

In this section we will present the results of the game evaluation. We will start by describing the methodology used to assess the impact of the game. The following two sections describe the results of the two questionnaires that were answered by the participants.

Methodology and analysis techniques

To test the gameplay experience and effects towards children’s perception about air pollution, formal summative tests sessions were conducted throughout the month of October in an elementary school located in the county of Sintra, in Portugal. We asked participants to perform an air pollution questionnaire before and after playing the game. The statements in this questionnaire were reviewed and validated by an environmental expert. We also performed a game experience questionnaire to assess gaming experience while playing the game. Participants played the game on a laptop, using as input interface either a mouse or the laptop’s touchpad, depending on the participant’s preference. A photograph of the test setup can be seen in Fig. 13.

Fig. 13
figure 13

Test setup

Full size image

Twenty elementary students aged between seven and nine years old participated in the testing sessions. Eleven of these students (55%) self-reported male and the remaining nine female (45%). The test sessions began by explaining the participant what the purpose of the test and what the participant would be doing. After explaining the testing procedure, the participant was asked for permission to begin the formal testing if she/he desired to continue. First, the participant was asked to fill a pre-game questionnaire about air pollution to evaluate one’s perception before playing the game. Then, the participant was asked to engage on a 25-minute gameplay session to experience the game. Right before playing the game, a summary of the game in terms of story, characters, goals, and controls was given to the participants. Participants also visualised a game tutorial to have a better comprehension of the controls and goals. After playing the game, participants were asked to fill a post-game questionnaire about air pollution, identical to the pre-game, allowing us to assess the learning gain resulting from playing the game. Lastly, participants were asked to fill a game experience questionnaire to evaluate the gaming experience overall.

As mentioned, the air pollution pre-game and post-game questionnaires are identical. In the first part of the questionnaire, the participant was asked to classify several statements regarding air quality using a five-point Likert scale, ranging from 1 (Completely Disagree) to 5 (Completely Agree). In the second part of the questionnaire, participants were asked to match air quality levels to AQI colours (see Table 1). Finally, in the third part of the questionnaire, participants were asked to answer a set of open-ended questions regarding indoor and outdoor air pollution. All answers were audio recorded with participants’ explicit permission. An english-translated version of the questionnaire is presented in Appendix 1 (the original is in Portuguese).

The mean (M) five-point Likert scale response across all participants was computed per statement, separately for the pre-game, \(M_b\), and post-game, \(M_a\), questionnaires. Matching between air quality level and AQI colour was also averaged across all participants, separately for the pre-game and post-game, by assigning one point for each correct answer. This resulted in two means. We can compare these two means to assess the impact of our game on children knowledge of air pollution (for each of these questions). Since we do not know the population mean and variance, and the sample size is small (\(n=20\)) we need to use the t-test. The null hypothesis is that the mean of the two populations (pre- and post-game) is the same. The alternative hypothesis is that the means are different, i.e., there is an impact of our game on children’s knowledge of air pollution. The degrees of freedom is 19. In the following sections, we report the result of the t-test, the p-value and the sample size.

By assigning different points to the Likert items and one point for each correct answer in the multiple choice grid question, we can quantitatively analyse the answers to these questions. Regarding the open-ended questions, we analysed the audio recordings to look for common keywords or concepts mentioned by the participants. Since many concepts were only referred once, we decided to only perform a qualitative analysis of these results.

As for the game experience questionnaire, this was only filled by the participant after playing the game. This questionnaire was purposely elaborated in a previous work as a questionnaire for evaluation of serious games towards children (Fernandes, 2021). The questionnaire is based on SUS (Brooke, 1996) and TAM (Davis, 1989), and has also been adjusted for children (Putnam et al., 2020). This questionnaire consists in nine statements to be classified by the participant with a five-point Likert scale ranging from 1 (Strongly Disagree) to 5 (Strongly Agree). These statements have a positive or negative bias which influences how the answer is used to calculate a score for all the nine statements. This questionnaire evaluates experience elements as enjoyment, intuitiveness, ease of use, perception regarding game’s value as an alternative to traditional teaching methods, perception of learning gain, and likelihood of game recommendation to a friend. The questionnaire also includes two open-ended questions, in which participants are asked about what they enjoyed the most and what they disliked the most in the game. The answers to these questions were audio-recorded using a mobile phone and the audio files were analysed afterwards. Lastly, the final section of the game experience questionnaire included questions about basic participants’ information, including age group, gender, and gaming frequency overall. After conducting the tests, answers for each item were analysed and usability scores were calculated using the SUS score technique for the nine Likert items (Lewis & Sauro, 2017). An english-translated version of the questionnaire is presented in Appendix 2.

Air pollution questionnaires results

Likert items

Results regarding the first statement of the air pollution questionnaire (all air pollution is visible) show that most participants did not agree or disagree and a few of them agreed in the pre-game (\(M_b=3.40\)), whereas in the post-game the mean shifted to a more negative opinion, but mostly stayed neutral towards the statement (\(M_a=2.90\)). Practically, it was significant enough to reject the null hypothesis at \(90\,\%\) level of confidence (\(t=-1.70, p=0.106, n=20\)). For the second statement (there is air pollution inside our homes), participants showed a slightly negative but almost neutral opinion towards the phrase in the pre-game questionnaire (\(M_b=2.75\)). After playing the game, participants agreed more towards the statement (\(M_a=3.75\)). The pre- and post-game responses are significantly different for a \(95\,\%\) level of confidence \((t= 2.44, p=0.025, n=20)\). In the third statement (the use of scented candles and air fresheners contributes to good air quality inside our homes), participants had a slightly positive opinion towards it before playing the game (\(M_b=3.50\)). After playing the game, participants opinion towards the phrase slightly decreased (\(M_a=3.35\)), showing no significant difference between pre- and post-game conditions \((t = -0.57, p=0. 577, n=20)\). Regarding the fourth statement (the use of cleaning products contributes to good air quality inside our homes), before playing the game, participants overall stayed neutral in their opinion towards the phrase \((M_b=3.20)\), which barely changed to a more negative opinion after playing the game \((M_a=3.15)\), showing no significant difference between pre- and post-game conditions \((t= -0.14, p=0.893, n=20)\). Finally, regarding the results obtained for the last sentence (the existence of gardens and vegetation near our houses can improve air quality in the area), participants showed a highly positive opinion before playing the game (\(M_b=4.55\)), whereas after playing the game, surprisingly, participants expressed a less positive opinion (\(M_a=4.10\)), showing a significant difference between pre- and post-game conditions for a \(95\,\%\) level of confidence (\(t =-2.27, p=0. 035, n=20\)). All these results are listed in Table 2.

Table 2 Results of the statistical tests applied to the statement classification questions of the air pollution questionnaires
Full size table

Multiple-choice grid questions

As for the mean results from the multiple-choice grid question about the colours that are associated with pollution levels, a slight increase occurred between pre-game \((M_b=1.85 \pm 1.24)\) and post-game \((M_a=2.30 \pm 1.49)\) though not significantly enough \((t=1.31, \, p=0.206, \, n=20)\).

Open-ended questions

When participants were asked if they think there could be air pollution inside our homes, in the pre-game questionnaire, half of the participants responded no (\(N=10\)), over a third responded yes (\(N=7\)), whereas the remainder did not know or did not respond. After playing the game, more than half of the participants responded yes (\(N=13\)), a quarter of the participants responded no (\(N=5\)), one participant did not know, and one reported that there was "more or less" air pollution inside our homes. Figure 14 shows the comparison of answers between both pre-and post-game. Hence, these results indicate that the game allowed participants to better acknowledge that the air in their places can be polluted.

Fig. 14
figure 14

Answers to open-ended question Do you think there is air pollution inside our homes?

Full size image

When participants were asked about what are the sources of air pollution inside their homes, several topics were addressed in their answers. In the pre-game questionnaire, the topic that stood out was garbage and other types of waste, such as plastic, cardboard and glass (\(N=10\)). Some participants did not answer or did not know any possible source (\(N=4\)). Some other terms were also mentioned, such as smoke, candles, cleaning products (\(N=3\)). The remainder of terms that were indicated were mentioned only once (\(N=1\)). After playing the game, despite being mentioned fewer times, garbage and other types of waste were still the most mentioned topics (\(N=7\)). Two of the topics got more mentions, namely, cleaning products (\(N=5\)) and candles (\(N=4\)), indicating a valuable learning gain. Smoke was mentioned the same number of times as in the pre-game (\(N=3\)). There were topics that were mentioned only once (\(N=1\)), mostly during the pre-game. Figure 15 shows the list of topics and how frequently they were mentioned in both questionnaires.

Fig. 15
figure 15

Answers to open-ended question What do you think can pollute the air inside our homes?

Full size image

Regarding their opinion on what is in polluted air, the pre-game questionnaire shows most participants mentioning garbage (\(N=11\)). Smoke and germs were mentioned the same number of times (\(N=3\)). A small number of participants did not respond or did not know (\(N=3\)). The remainder of topics were mentioned only once (\(N=1\)). After playing the game, post-game questionnaire results show that the topic garbage was mentioned fewer times (\(N=8\)), there was an increase in number of mentions for the topics smoke (\(N=6\)), germs (\(N=4\)) and bugs (\(N=2\)), and there were less people who did not respond or did not know (\(N=2\)), indicating an interesting learning gain. There were many topics that were mentioned only once (\(N=1\)) primarily in the pre-game. Results can be seen in Fig. 16.

Fig. 16
figure 16

Answers to open-ended question What do you think is in the air when it is polluted?

Full size image

Regarding their opinion on how people can mitigate outdoor air pollution, in the pre-game questionnaire, the topics more mentioned were cleaning or picking up garbage (\(N=8\)), stop polluting and less factories (\(N=2\)). Some participants did not respond or did not know (\(N=5\)) and other topics were mentioned only once (\(N=1\)). In the post-game questionnaire, a higher number of suggestions were given by the participants, the most suggested action being picking up garbage again (\(N=7\)). The number of participants who did not know or did not answer remained the same as in the pre-game (\(N=5\)). A higher number of participants suggested to not pollute the environment (\(N=4\)) and to have more gardens (\(N=2\)). There were also new topics with increased use of bicycles being the most mentioned (\(N=2\)), possibly a direct result of the main game mechanic, i.e., driving a bicycle around the city. Figure 17 shows the answers in the pre- and post-game for this question.

Fig. 17
figure 17

Answers of open-ended question How do you think we can reduce the air pollution we have outside?

Full size image

Game experience questionnaire

Likert items

The results on the game experience questionnaire indicate an overall positive gaming experience. Participants wanted to continue playing the game (\(M=4.35\)). They would like their teacher to use this kind of game in classroom and they also agreed that they would play it more if they had this game at home (\(M=4.25\)). Participants agreed that they learned a lot about pollution when playing this game (\(M=4.55\)) and slightly agreed that they felt confused several times when playing the game (\(M=3.55\)). They tended to be neutral (\(M=2.95\)) about their need for help from an adult while playing and they believed that they would learn more about pollution if they played this game more times (\(M=4.65\)). Although participants only slightly agreed that their friends would enjoy playing this game (\(M= 3.85\)), they agreed that their friends would learn more about pollution by playing this game (\(M=4.60\)). These results, which are largely positive, are summarised in Table 3. A positive experience while playing the game is important as it is an indicator that children may wish to spend time (re)playing the game and, thus, learning about air pollution.

Table 3 Results of the game experience questionnaire using a five-point Likert scale with M and STD standing for mean and standard deviation, respectively
Full size table

As for the usability score, scores ranged from 94 to 53, with a mean of 75 out of 100. The game experience questionnaire is an adaptation of a SUS questionnaire. In Sauro (2011) the author has examined several studies using the SUS questionnaire and has found out that a threshold of 68 is the borderline between usable and non-usable applications. While our questionnaire is an adaptation of SUS and we lack the broad study of Sauro (2011), we consider the score 75 that we obtained as acceptable.

Open-ended questions

Figures 18 and 19 show the results of the two open-ended questions about what participants liked the most and the least about the game, respectively. Regarding positive aspects, the most frequent answers were the sensor mini-game (\(N=11\)) and controlling the bike and the dilemma mini-game (\(N=7\)). There were other less frequent answers, such as learning new things about pollution (\(N=4\)) and the dialogue (\(N=1\)). As for the negative aspects, results show that about half of participants had nothing to report (\(N=8\)). The aspect that received the highest number of complaints (\(N=7\)) was the way participants had to control the bike. Two participants complained about the dialogue for being too long. Remaining aspects, such as too many things to do, placing sensors to answer dilemmas, and the guide feature that kept trying to guide the player, were mentioned only once (\(N=1\)). Overall, the raised issues are easy to overcome in a future version (e.g., bike controls) or improved by means of editing the resource files (e.g., dialogues clarity).

Fig. 18
figure 18

Frequency of positive aspects mentioned by participants in the open ended question “What did you like the most about the game?” asked during the game experience questionnaire

Full size image
Fig. 19
figure 19

Frequency of negative aspects mentioned by participants in the open ended question “What did you like the least about the game?” asked during the game experience questionnaire

Full size image

Discussion

Air pollution questionnaires

As mentioned, the pre- and post-game air pollution questionnaires are identical. Since the pre-game questionnaire is filled before children see the content of our game, this means that we can consider the results of this questionnaire as a base line or control group data in that it represents the knowledge the average child has on air pollution. If we compare with the results of the post-game questionnaire, we can check if there is a change in participants’ perception of air pollution.

Comparing the results between the pre- and post-game questionnaires, we can see that in three out of five statements there was an improvement in the knowledge of air pollution (the differences are statistically different). In particular, students showed improvements regarding their ability to acknowledge the potential invisibility of all air pollution and the presence of air pollution inside home (statements 1 and 2 in Table 2 and in appendix section 1.1). This is particular important given that particulate matter is a relevant pollutant, even in indoor environments (Balasubramanian et al., 2010).

Interestingly, after playing the game, children’s belief that the presence of gardens and vegetation near their houses can improve air quality, decreased. This was unexpected given that the in-game dialogues associated vegetation, trees, and parks to good air quality (Diener & Mudu, 2021). Perhaps these results are indicating that, after playing the game, participants realize that planting trees and building parks alone is not enough to improve air quality, that other actions must be orchestrated as well. This hypothesis needs to be validated in future work.

There was no observable statistical difference in the statements mentioning scented candles, air fresheners or cleaning products. One reason this occurred may lay on the fact that some players did not reached the sensor and dilemma mini-games levels that address this issue. Hence, future versions of the game need to ensure that the player gets in contact with every key content regardless of their skill level. These statements are related to indoor sources of air pollutants which children should be aware of (Manoukian et al., 2013).

In the open-ended questions, when participants were asked if air pollution exists inside our homes, half of them said no before playing, whereas after playing the game more than half of the participants said yes. This shows that there were positive effects on participants’ perception about indoor air pollution. Moreover, it was also observed an improvement regarding children’s perception about indoor air pollution, indicated by the higher frequency that cleaning products and candles were mentioned after playing. In addition, more children also managed to answer the question related to this topic, which is also a positive result. The results also show that the game managed to increase some of the participants’ perception regarding air pollution, indicated by the higher frequency that terms as smoke and germs were mentioned after playing the game. Interestingly, even though the game never addresses the germs topic, an increased number of answers relating germs and air pollution was observed after playing the game, possibly indicating a cross-learning effect. The highly frequent mentions to garbage in children’s answers suggest that most participants conceive bad odours and dirt as important cues of poor air quality and high pollution.

For the last open-ended question, participants gave their opinion about what actions they think are needed to reduce air pollution outdoors. The pre- and post-game questionnaires results show that most participants suggested actions as picking up trash. Importantly, in the post-game, results also show that there was an increase in the number of participants suggesting to stop littering. There were also more answers about adding more gardens and vegetation. New suggestions have also come up where some participants suggested an increase in the use of bicycles as a mean of transport. Overall, results show that after playing the game, children are able to articulate more terms related to pro-environmental behaviours, which indicates that the game improved participants’ perception regarding air pollution and air quality.

Game experience questionnaire

The results obtained with the game experience questionnaire were used to compute a usability score inspired by SUS. Given that the questionnaire has nine items rather than ten items, as it is the case of standard SUS, the computed usability score was adjusted according to Lewis and Sauro (2017). Although the applied questionnaire mixes items from SUS and TAM, the contribution of each item to the usability score was accounted for as done in SUS, that is, distinguishing items with positive and negative biases. We consider a threshold of 50 in the computed usability score (the score ranges from 0 to 100) to classify a game as usable, which is above the obtained 75.

The open-ended questions about what participants liked and disliked the most were also analysed and the obtained results show that participants liked the sensor mini-game the most. Nevertheless, some of them also appreciated the dilemma mini-game and controlling the bike. As for the negative aspects, several of them did not like the controls that were implemented. Participants might have become confused with the orientation of the streets, despite the control keys being aligned for each possible direction. We noted that some players took some time to get used to it, which may have affected their progress in the game. Controls need to be more intuitive in the future so participants can have a better gameplay experience right from the start. Few participants also mentioned that there was too much dialogue. This excessive dialogue would make some participants skip some dialogue midway through their gameplay session. Other participants would read most of the dialogue but some of these elementary students have more difficulties in reading than others and, therefore, some of them spent a considerable time of the gameplay reading the dialogue lines and progressed in a much slower pace.

Conclusions and future work

This article presented Problems in the Air, an isometric serious game developed in Unity whose main goal is to improve children’s awareness regarding air pollution and air quality. A distinguishing trait of the presented game, in comparison with previous serious games in the topic, is the focus on the engineering aspects of sensor-based air quality monitoring. This way, in addition to raise environmental awareness, the game also intends to promote the appeal of engineering and problem solving in the environmental sciences area. The game has been devised so that it can be easily tailored and expanded by educators and teachers, through editable XML resource files, allowing them to match children’s background and age group. This customisation includes new dialogues, new missions, and new dilemmas.

The game was tested with twenty elementary students between seven and nine years old. Statistical tests show that in three out of five questions there was an increase in children’s perception about air pollution. However, the generalisation of these results may be limited by the small sample size. Therefore, further tests should be conducted in order to increase the statistical power of the game impact’s results.

In terms of usability, the employed sample size is sufficient to identify interface shortcomings. In our case, results showed that children enjoyed playing the game and would like to use the game more often. Using the SUS score technique, the mean usability was 75 out of 100, which can be considerable acceptable, as it is well above the scale’s midpoint.

The answers to the open-ended questions of the air pollution questionnaire produced a variety of concepts. They reflect personal experiences and can also be subjective. In a long term study (one or two year duration), the answers to open-ended questions can be used in follow up questionnaires in true or false statements in order to check the knowledge children possess. Another possibility is to analyse the answers and check if there are concepts that children are unaware of in order to create new game content, thus taking advantage of the game customisation capabilities.

Answers to open-ended questions of the game experience questionnaire raised some issues that will be addressed in the future. It was found that the game’s controls need to be more intuitive. To cope with this issue, the game can be adapted to be deployed on mobile phones and tablets, facilitating the design of touch-based interactions. Moreover, better situational awareness and vehicle controls can be borrowed from existing video games to which players are used to. It was also found that the wording of the dialogues must be improved to ease reading and avoid boredom. This can be resolved easily with a focus group of teachers tasked to adapt the XML resource files.

Finally, the customisation capabilities of the presented game can be exploited to go beyond air pollution, for instance, by also addressing soil and water pollution. The current city map can be used to explore soil pollution, but it would be more pedagogical if the player could travel to the forest or the countryside, instead of being restricted to roads. As for exploring water pollution, it would benefit from a map with a river or a coast side, where the player could place sensing devices to measure water quality. In both cases, as we have said, the content of available mini-games can be edited by educators and teachers to provide explanations on why a river stream or a plot of land has high pollution.

Gamification is a technique that is used to teach new concepts and foster better behaviours. However, effects tend to be short-term. The goal of this purpose was on presenting Problems in the Air. Having laid the foundations of this tool, we can on future work focus on a longitudinal study and follow a group of children and track their behaviour during the course of one to two years. Given the customisation capabilities, different aspects of air pollution can be addressed and checked through corresponding air pollution questionnaires. This way we can study the long-term influences of using gamification to evaluate children’s knowledge of air pollution and to foster environmental friendly behaviours.