Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Adapting the Complexity Level of a Serious Game to the Proficiency of Players

Adapting the Complexity Level of a Serious Game to the Proficiency of Players As games are continuously assessing the player, this assessment can be used to adapt the complexity of a game to the proficiency of the player in real time. We performed an experiment to examine the role of dynamic adaptation. In one condition, participants played a version of our serious game for triage training that automatically adapted the complexity level of the presented cases to how well the participant scored previously. Participants in the control condition played a version of the game with no adaptation. The adapted version was significantly more efficient and resulted in higher learning gains per instructional case, but did not lead to a difference in engagement. Adapting games to the proficiency of the player could make serious games more efficient learning tools. Keywords: dynamic adaptation, engagement, learning efficiency, proficiency, serious game Received on 29 December 2013, accepted on 05 March 2014, published on 22 May 2014 Copyright © 2014 H. van Oostendorp et al., licensed to ICST. This is an open access article distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/3.0/), which permits unlimited use, distribution and reproduction in any medium so long as the original work is properly cited. doi: 10.4108/sg.1.2.e5 [4], and some learners will therefore benefit from a slower 1. Introduction pace in the presentation of instructional material in order to correctly organize all the new information that is coming in. Serious games can be used to engender learning in a player, Conversely, efficient learning may also be hindered by and two recent meta-analyses have shown that the usage of cognitive underload, where the learner is stimulated too serious games may even lead to superior learning compared little, for instance when a quick learner plays a game that with traditional (but passive) instructional methods [1, 2]. has a slow pace in order to accommodate slow learners. However a serious game is found to be primarily efficacious Cognitive underload can lead to (passive) fatigue, which has if a person is allowed to play the game multiple times [1], a been shown to result in disengagement from the task and result that Wouters and Van Oostendorp [3] argue higher distractibility and can subsequently degrade underlines the notion that games are complex environments performance [5, 6]. If a game were to actively prevent the in which the player first has to learn how to control the player from becoming cognitively overloaded or game and the way in which it conveys the instructional underloaded, it could therefore be more efficient [7]. material, before this material itself can be learned. Games in Secondly and closely related to this, Csikszentmihalyi turn are products that have to be made beforehand and have [8] posited that one can experience the feeling of flow, a preset pace, and often do not take into account the which is a feeling where someone is completely engaged in individual learning rate. an activity to the point of losing self-consciousness and the People learn at different speeds, which may lead to a activity becomes rewarding in its own right, and that this number of problems. Firstly, the rich multimodal leads to the individual functioning at his or her fullest information of a game may overload the limited working capacity [9]. This is achieved when the provided challenge memory capacity of a player, leading to incorrect learning is optimally suited to the skills of the user; and as ___________________________________ videogames are often stated to be engaging, with players Corresponding author : h.vanoostendorp@uu.nl European Alliance EAI Endorsed Transactions on Serious Games EAI for Innovation 04 2013 - 05 2014 | Volume 1 | Issue 2 | e5 H. van Oostendorp et al. reporting an experience of being completely absorbed in the (serious) games are developed, and progress has been made, game, they seem to be ideally suited to produce flow [10, empirical research to effects of adaptivity in terms of 11]. Flow has been shown to be positively correlated to learning and engagement are still scarce [see also 20].. In learning [12]; therefore, keeping players in a sense of flow this paper we will remedy this and present results of an by adjusting the challenge to their skills could improve empirical study on the learning and affective effects of a learning [13]. game with dynamic adaptivity. That is, a game where the Summarizing, if quick learners were able to progress in challenges of, or difficulties caused by, the game are the game at a faster pace, for instance because the game increasing, and at a rate dependent on the proficiency of the recognizes their proficiency and adapts the game player (online adaptation). We will mainly be concerned accordingly, engagement in performing the task could be with varying the attributes of the non-player characters. enhanced which in turn results in a higher efficiency of the game. Similarly, a slower pace for slower learners would 2.2. Assessing the proficiency of players also improve engagement and efficiency for them. In this paper we will examine in an experimental study whether For the principle of fitting the instruction to the learner's adapting a serious game to the proficiency of players proficiency level to be implemented in serious games improves learning and engagement. But first we will, in the effectively, it is important first that the proficiency should next section, discuss different aspects of adaptivity in be assessed and secondly that the challenge should be general, how we monitored or assessed proficiency of adapted to the player automatically in a non-obtrusive way. players and how we implemented adaptivity in a dynamic Automatically assessing and adapting the challenge or way in the serious game Code Red Triage. difficulty of a game to the proficiency of a player is slowly becoming commonplace in entertainment games. For instance in Rocksmith [21], a musical instrument simulation 2. Adaptivity game, the player needs to hit the correct notes of a song with good timing. The game adds more notes and places a greater 2.1. Aspects of adaptivity emphasis on timing when the player performs well, or vice versa when the player performs badly. Racing games like In line with Lopes and Bidarra [14], we can distinguish Mario Kart [22] and Need for Speed [23], implement a several components of adaptation. 1) The game world and simple adaptation known as ‘rubber banding’: when the its objects can be varied, e.g. the layout of the game world player lags behind the other racing contestants, they will can be made simpler for underachieving players [15]. 2) The slow down in order to let the player catch up with them – game play mechanics, how game elements work, including when the player is up front, his opponents will become actions like running or shooting, e.g. adjusting shooting faster and try to keep up with him. difficulty by providing player aim assistance, according to Here, we will elaborate on two modes of assessing that individual skills [16]. 3) Adapting the attributes of the non- are most relevant to our research. Firstly, one interesting player characters in the game, e.g. increasing the abilities of avenue in which a game can be adapted to the player was the non-playing character when the player performs well. undertaken by Yun et al. [24], who used an infrared camera Domain knowledge is here automatically gathered by the that was mounted on a TV displaying the game. This camera game based on Artificial Intelligence-techniques, in order to (overtly) recorded the faces of the participants while they offer more challenging behavior of the non-player characters were playing a game that revolved around shooting enemy [17]. 4) Game narratives, e.g. adapting the sequence of robots. Looking at the heat signatures from the supra-orbital events to the pace or behavior of the player [18], and 5) region of the face, they were able to derive how much game scenarios - more or less similar to the previous one: apparent stress the game exerted on the player during game adapting the flow of events and actions within a game, that play. At the same time, the player reported at set intervals is, adapting the progression within a game level to the whether they found the game too easy, just right or too learning goals of the player. For instance, monitoring the difficult, and whether they were enjoying the game or would players actions and based on that certain points in the plot like to quit. This research is relevant to our own for two are included in the game (or not) [19]. reasons. One, they discovered that people who found the A next issue in creating adaptive games is to decide on game too difficult and wanted to quit actually had lower the method of generating the content. Lopes and Bidarra stress levels than when the game was moderately difficult. distinguish two general methods. First, offline adaptivity (or They argued that this is due to the player becoming customized content generation); adjustments are made disengaged with the game, thereby corroborating the considering player-dependent data, but prior to initiating the previously made assertion that too high a challenge leads to gameplay. Secondly, online adaptivity, i.e. adjusting the cognitive overload and is detrimental to the engagement or game to its players, in real time, as they play. flow experience. Two, a version of the game where the A further discussion on the way adaptation can be game automatically assesses and adapts to the stress level of implemented in games and the associated challenges can, for the player was shown to lead to higher engagement and instance, be found in Lopes and Bidarra [14]. Though in the better in-game performance (in terms of how many robots (game) industry and academia now many different adaptive European Alliance EAI Endorsed Transactions on Serious Games EAI for Innovation 04 2013 - 05 2014 | Volume 1 | Issue 2 | e5 Adapting the Complexity Level of a Serious Game to the Proficiency of Players were defeated) than in conditions with preset difficulty attention. The mobility (sieve) triage taught here is a levels, even for the easy difficulty level. relatively simple procedure, where it takes the first Another interesting example of how to adapt the game to responder between one and five steps to determine the the player is the entertainment game The Elder Scrolls 4: severity of the victim’s injuries. When the game starts, the Oblivion [25]. Here, the player roleplays a character in a player finds himself in an empty train station with signs of large and open medieval fantasy world. As the player recent panic. Here, he learns that he is a medical first encounters new locales, performs quests and defeats responder who has received a call that a bomb has gone off monsters, his or her character will gradually become on a subway platform. The player is then told to find the stronger and gain better weapons and items (see further subway platform and perform the triage procedure on the Shute et al., [26]). Because the game features an open world victims. Upon reaching the subway platform (see Figure 1), for the player to explore freely, this traditionally leads to a visible timer starts counting down from seventeen minutes. problems where the player may encounter monsters that are When the timer reaches zero, the game ends. This timer was far too strong for his or her avatar to defeat at that point in added to instill a sense of immediacy and stress; in practice time. To counter this and provide the optimal experience for almost every participant is able to triage all victims everyone, the player’s adversaries in the game also progress comfortably within this time. At the subway platform, the in power at the same rate as the skill level of the player. player can then walk up to a victim and press a button to Contrarily to what would be expected, many gamers enter the triage menu, which consists of eight buttons for criticized this feature, as it made them feel that their actions triage actions, and four buttons for the four different triage were largely inconsequential [27]; they were not getting categories (see Figure 2). Pressing a triage button will give a stronger than their enemies and therefore they didn’t feel few lines of general information on what the action entails like they were mastering the game. and approximately at what stage in the procedure it should Above we mentioned two different techniques of be used, and a line with specific information on how the assessing the player proficiency within the game. The first action affected the victim the player’s looking at. After was a more overt technique, where in real life settings the choosing a few triage actions the player should be able to player would have to install an infrared camera for it to have an idea how heavily injured the victim is and assign a work; the second example featured so-called ‘stealth’ triage category. assessment [7, 26], that is, a more covert assessment that is coupled to the naturally occurring moves of the player in the game. In essence, all games are an assessment device, in that progressing past an obstacle is contingent on acquiring the needed knowledge of how to do so. As digital games are played on computers, which require that every game rule and in-game problem encountered is computable, determining whether the player succeeded is often easily quantifiable. 2.3. Dynamic adaptivity in the serious game Code Red Triage As indicated we want to study whether the online adaptation of the challenge or difficulty of a learning experience to the proficiency of players, improves learning and enhances engagement. Following [28] we use the term dynamic adaptivity to designate online adaptation of game experiences in terms of complexity and matching that to the proficiency of players. In order to test this hypothesis we used the serious game Code Red Triage, a total conversion mod of Half-Life 2 [29-31]. The game is designed to teach Figure 1. Subway platform in the game Code Red the triage procedure, a procedure for medical first Triage responders to prioritize the victims of a mass casualty event according to how urgently the victim needs medical European Alliance EAI Endorsed Transactions on Serious Games EAI for Innovation 3 04 2013 – 05 2014 | Volume 1 | Issue 2 | e5 H. van Oostendorp et al. Figure 2. Triage menu in the game Figure 3. Feedback after categorizing a victim in a triage category Once this is done, the victim changes color to depict the operationalized as the game deleting all remaining victim chosen category and the player receives a score showing cases within the same tier, if the player scored higher than how well he did, as well as a few lines telling him whether a threshold value for that victim. The threshold was or not a) he forgot to take procedure steps, b) took steps in determined with the data from a pilot experiment, by the wrong order, c) took unnecessary steps and d) whether rounding up the average score per victim tier. A player it was done within the allotted time (between 10 and 55 who was unable to triage a victim case and scored below seconds), see Figure 3 for a screenshot. The in-game score the threshold, received one or more of the remaining cases that can be obtained per victim ranges from 0 to 100 and of that tier before going to the next level of complexity. In is based on the previous four criteria. other words, more successful players could attain the most In the case of Code Red Triage, we already have a complex case in less cases, and consequently learn to measure to assess how well the player is performing in the perform the triage more efficiently. In the control version game, namely the in-game score, which provides us with of the game all (19) cases were presented in a gradually an objective measure of whether the player is able to increasing complexity. correctly apply the procedure to a given victim case. The We hypothesize that players feel more engaged by the player’s performance can therefore be seen as an dynamic adaptive version, because the game always indication of their proficiency level [7]. We can thus use remains challenging (compared to a control version), and the above mentioned covert method to assess the secondly we expect in the dynamic adaptive version of the proficiency of players here. game that players are able to learn more efficiently, We used this in-game score to adapt the difficulty of because redundant learning experiences (triage cases) can the game to the proficiency of the player. In Code Red be skipped. Triage, there are a total of six paths with an increasing number of steps in the triage procedure that are taught 3. Method with the game, but there are multiple victims for any given path. As the victims are encountered in increasing order of complexity (i.e. the number of steps needed to 3.1. Participants come to a correct categorization), these groups of victims are called ‘victim tiers’. In the set of victims 6 tiers or In total 28 individuals of university-level education, 19 levels of complexity were distinguished. In other words, male and 9 female, participated in the experiment, and the attributes of the non-player characters were varied in were randomly assigned to the adaptive game condition complexity. If a player scores above a preset threshold, he (n=14), and the control condition (n=14). Average age or she has proven to have a certain level of proficiency was 22.86 with a standard deviation of 5.68. and can move on to a more complex victim tier. In the adaptive condition of Code Red Triage this was European Alliance EAI Endorsed Transactions on Serious Games EAI for Innovation 04 2013 - 05 2014 | Volume 1 | Issue 2 | e5 Adapting the Complexity Level of a Serious Game to the Proficiency of Players 3.2. Materials 3.3. Apparatus and procedure To measure the learning of players, three types of instruments were used. The in-game score (see above) The game was played on a 17” laptop at a resolution of formed the first measure: an indication of the progression 1920 x 1200 with circum-aural headphones in a room with of the player in the game. In several studies done with the the lights turned off. The graphics settings were set at same game and the same in-game score we found that the their maximum and the game ran at a constant 60 frames in-game score significantly correlated with a knowledge per second. The participants were asked to perform the test presented after the game [31, 32], which gives structural knowledge assessment with the PCKNOT plausibility to the notion that the in-game score, conceived software. Then, the knowledge test was administered. as analytical learning tool [33], is a valid measure of Before playing the game, the participants were given learning. Statistics from the game that were logged instructions about Code Red Triage and were informed furthermore included triaged victims, number of triaged about its goal. Nothing was revealed to them about the victims, tier of victim, time per victim, total time, score condition they took part in. Playing the game from start to per victim and total score. Second and third, we measured finish took each participant at most 25 minutes: a few how much a participant learned in the game with two minutes for the entry level, a few more for the hallway measures: a pen-and-paper knowledge test and a structural part and a maximum of 17 minutes was allowed for the knowledge assessment. The knowledge test was in the metro platform part, in which the triages took place. The form of eight verbal and eight pictorial multiple choice scores participants reached in the game gave information questions where the player had to answer questions related about their performance (see also section 2.3). Directly to the triage procedure by choosing one of four after the participants finished playing the game, they were alternatives (total score range 0-16). asked to fill out the engagement questionnaire. They were Whereas the knowledge test measured how well the then asked to do the structural knowledge assessment and participant could reproduce declarative knowledge, the knowledge test as before, but with the questions in a structural knowledge assessment determined how the different order. Finally, the participants were thanked for information was organized on a deeper, more structural their cooperation and they received a coupon for their level. Here, a computer program called PCKNOT [34] was work. An overview of the procedure can be seen in Figure used, that let participants rate the degree of relatedness of pairs of concepts from the triage procedure. These ratings could subsequently be used to elicit a participant’s knowledge structure with the Pathfinder metric [35] and compared to the knowledge structure of experts; resulting in a similarity measure that indicated how well the participant had organized the information of the triage Figure 4. Procedure of the experiment procedure structurally [36]. The score range varies from - 1 through 0 to +1. Pathfinder has been successfully applied by [37] to measure learning from a complex videogame. They found that it was also predictive of skill 4. Results retention and skill transfer. For further information see Engagement Wouters, Van der Spek and Van Oostendorp [38]. In our case we focused on 8 important concepts from the triage The mean scores and standard deviations of the procedure and consequently 28 pairs were presented for engagement questionnaire are mentioned in Table 1. An the related judgments. The created networks were ANOVA showed no significant effect of condition on the compared with the referent structure that was derived by ITC-SOPI engagement questionnaire, F(1,26) < 1. averaging the elicited knowledge structures of the current researchers. Learning Efficiency The engagement of players was measured by using the subscale of the ITC Sense of Presence Inventory (ITC- There are several ways to determine whether learning was SOPI), which indicates the participant’s feelings of more efficient in the adaptive condition. A reliable engagement with a twelve item five-point Likert scale measure for efficiency is to divide the posttest scores of [39]. If the challenge of the game is better adjusted to the the participants by the number of victim cases triaged, abilities of the player, one would expect the player to be giving us an indication of how much the participant has drawn into the game more, which we hoped to see learned per unit of instruction, and whether this would be expressed in the scores on this subscale. The reliability of higher in a game that adapts the information presentation the ITC-SOPI Engagement questionnaire appeared to be to the player’s proficiency. Another way would be to relatively low, Cronbach's coefficient α = 0.59. divide learning performance by total time spent playing the game. However some players navigate more efficiently than others towards the platforms etc, which European Alliance EAI Endorsed Transactions on Serious Games EAI for Innovation 5 04 2013 – 05 2014 | Volume 1 | Issue 2 | e5 H. van Oostendorp et al. blurs what we want to measure. We therefore decided to point. The intervention itself may be too small next to all use learning performance divided by the number of cases the other determinants of engagement, such as the game’s triaged, as a purer measure of learning efficiency. setting, world, expectations, control interface, et cetera, to show up as a difference on the rating scale, but the adaptive version may still be preferred when the conditions were placed side by side. A second explanation could be related to the fact that we only asked participants Table 1: Mean engagement and efficiency scores on to appraise their engagement after the game. It is unclear knowledge test and structural knowledge assessment whether a continuous measurement of a participant’s (sd). engagement, for instance with an infrared camera as in the research by [24], as we mentioned in the introduction, would have resulted in higher ratings throughout the game Control Adaptive in the adaptive version. Thirdly, people may play games Condition Condition for different reasons; a higher challenge could lead to Engagement (1-5) 3.63 (.33) 3.66 (.45) higher engagement in some players, whereas it has the opposite effect on others. Lastly, and perhaps as a result Knowledge test .57 (.19) 1.02 (.30) of the previous explanation, we found that the Structural homogeneity of the engagement questionnaire knowledge .015 (.004) .028 (.019) (Cronbach’s alpha) was low. Perhaps this measurement assessment problem contributed to the fact that we did not find an effect of engagement. An ANCOVA with the pretest as covariate, condition as We saw that participants learned more per victim case fixed factor and posttest score divided by the total number in the adaptive condition compared to the control of victims triaged as dependent variable showed that condition. It could be that the moment a participant grasps condition had a significant effect on both the knowledge the procedure to resolve a victim case pertaining to a test (F(1,25) = 21.98, p < .001, d = 1.81) and the structural certain tier, the information presented in the following knowledge assessment (F(1,25) = 5.05, p < .05, d = .89). victims in that tier is redundant, at least to a point that it The means on these relative measures and standard does not improve learning of the procedure anymore, deviations of these tests are listed in Table 1. making the adaptive version more efficient. In order to determine whether the adaptive condition In-game score not only made learning the instructional material more efficient, but also leads to deeper learning [40], other The total in-game score was significantly higher for the experiments should be set up such as e.g. a study where control condition (M = 777.7, SD = 321.2) than for the learning is also measured after a longer delay or with adaptive condition (M = 316.4, SD = 107.8), F(1,26) = transfer tasks. However, some corroboration may be 25.95, p < .001, however this more or less follows from found in the structural knowledge assessments. They point the result that participants triaged significantly less to deeper learning in the adaptive condition. victims in the adaptive condition. One last observation concerns the relation between engagement and learning; the results found indicate that an increase in engagement does not seem necessary to 5. Conclusion and discussion enhance learning efficiency. Also the correlation between engagement and learning efficiency appeared to be low We hypothesized that a serious game that dynamically and not significant (p > .05) for both groups of adapts its challenge, or complexity presentation, to quick participants. However, for this finding too, the same learners could make a serious game more engaging and remarks as before should be made concerning the more efficient. The first part of the hypothesis was not measured engagement of players. confirmed, while the second part was confirmed; All in all, a rather simple alteration of a serious game participants in the adaptive game version learned where it dynamically adapts the presentation of significantly more per victim case than in the control complexity to the player’s performance and thereby its condition, and were therefore more efficient. challenge has been shown to markedly improve the We found no difference in the engagement ratings. If efficiency thereof. This is a promising result for serious the improved learning per unit of instruction was due to games developers that worry about the comparative less disengagement from the task, one would expect this efficiency of their game, as well as for researchers to appear from the results of the engagement interested in improving games with the aid of more questionnaire. We propose four explanations why we did sophisticated adaptation engines. It can also be a useful not find a difference in engagement. result for entertainment game developers, as many games Firstly, when participants had to appraise their need to incorporate tutorial levels that are necessary for engagement just after playing the game, they lacked players to understand the game, but are not a lot of fun to knowledge of the other condition and thereby a reference play, especially upon repeated playthroughs. A dynamic European Alliance EAI Endorsed Transactions on Serious Games EAI for Innovation 04 2013 - 05 2014 | Volume 1 | Issue 2 | e5 Adapting the Complexity Level of a Serious Game to the Proficiency of Players [3] Wouters, P.J.M., and van Oostendorp, H. (2013) A adaptive version that adapts to the player's proficiency meta-analytic review of the role of instructional could greatly speed up these mandatory instructional support in game-based learning. Computers & sequences and (possibly) make them more challenging. Education, 60: 412-425. [4] Moreno, R. and Mayer, R. (2007) Interactive Future research multimodal learning environments. Educational Above we already mentioned two limitations to our study, Psychology Review, 19(3): 309-326. viz. that it is impossible to conclusively state whether [5] Saxby, D. J., Matthews, G., Hitchcock, E. and Warm, dynamically adapting to the player’s performance only J. S. (2007) Development of active and passive fatigue manipulations using a driving simulator. In resulted in more efficient instruction, or also in deeper Proceedings of the Human Factors and Ergonomics learning, and that it is unclear whether participants Society 51st Annual Meeting (Santa Monica, CA: differed in engagement during gameplay. In addition, Human Factors and Ergonomics Society), 1237-1241. another limitation of our experimental setup that warrants [6] Paas, F., Renkl, A. and Sweller, J. (2004) Cognitive future research is that we did not measure retention over load theory: Instructional implications of the longer time periods. Participants in the adaptive game interaction between information structures and version received less practice and consequently less cognitive architecture. Instructional Science, 32(1): 1- opportunity to internalize the information. Therefore there is a real possibility - or even danger - that the participants [7] Shute, V. J. (2011) Stealth assessment in computer- based games to support learning. In S. Tobias & J.D. in the dynamic adaptive condition remember less of the Fletcher (eds.), Computer games and instruction instruction after several weeks. Regarding dynamic (Charlotte, NC: Information Age), 503-524. adaptation itself, in this study we did make some specific [8] Csikszentmihalyi, M. (1988) The flow experience and choices during the design and implementation process. its significance for human psychology. In M. We focused on the nature of the non-player characters and Csikszentmihalyi & I.S. Csikszentmihalyi (eds.), let them vary in number of steps needed to perform a Optimal experience: Psychological studies of flow in correct triage. Several alternatives are open for continued consciousness (Cambridge, MA: Cambridge research to the role of dynamic adaptation. For instance, University Press), 15-35. the set of buttons for executing the triage actions could be [9] Shernoff, D. J., Csikszentmihalyi, M., Shneider, B. and Shernoff, E. S. (2003) Student engagement in high adapted, that is, starting simple and increasing over time, school classrooms from the perspective of flow theory. depending on performance. Or the feedback given to School Psychology Quarterly, 18 (2): 158-176. players could be adapted, e.g. stating more or less [10] Sweetser P. and Wyeth, P. (2005) GameFlow: a model explicitly what went right and what went wrong while for evaluating player enjoyment in games. Computers performing the triage [41], or explanatory feedback could in Entertainment, 3(3): 1-24. be included, particularly in early learning phases [42]. [11] Cowley, B., Charles, D., Black, M. and Hickey, R. Finally, another option for making training procedures (2008) Toward an understanding of flow in video adapt themselves to participants is the notion of games. Computers in Entertainment 6(2): 1-27. adaptability. In this form of offline adaptation participants [12] Webster, J., Trevino, L. K. and Ryan, L. (1993) The dimensionality and correlates of flow in human- indicate themselves what direction they want to practice computer interactions. Computers in Human Behavior, and what part of the procedure they want to repeat. These 9(4): 411-426. are questions that still need to be examined in the future. [13] Liu, M., Horton, L., Olmanson, J. and Toprac, P. (2011) A study of learning and motivation in a new media enriched environment for middle school Acknowledgements science. Education Technology Research Development, 59: 249-265. This research has been supported by the GATE project [14] Lopes, R. and Bidarra, R. (2011) Adaptivity (http://gate.gameresearch.nl/), funded by the Netherlands challenges in games and simulations: a survey. IEEE Organization for Scientific Research (NWO). Transactions on Computational Intelligence and AI in Games 3(2): 85-99. [15] Walker, J. (2009) Left 4 Dead 2: Exclusive RPS Hand on Preview. Rock Paper Shotgun, PCGaming since References 1873. Online available: http://www.rockpapershotgun.com/2009/06/01left-4- [1] Sitzmann, T. (2011) A meta-analytic examination of dead-2-exclusive-rps-preview. the instructional effectiveness of computer-based [16] Brathwaite, B. and Schreiber, I. (2009) Challenges for simulation games. Personnel Psychology: 489-528. game designers, (Boston, MA: Charles River Media). [2] Wouters, P.J.M., van Nimwegen, C., van Oostendorp, [17] Bakkes, S., Spronck, P. and van den Herik, J. (2009) H. and van der Spek, E.D. (2013) A Meta-analysis of Rapid and reliable adaptation of videogame AI. IEEE the Cognitive and Motivational Effects of Serious Transactions Computational Intelligence and AI in Games. Journal of Educational Psychology, 105: 249- Games, 1(2): 93-104. [18] Roberts, D.L. and Isbell, C.L. (2008) A survey and qualitative analysis of recent advances in drama management. Int. Trans. Sci. Appli., 3(2): 61-75. European Alliance EAI Endorsed Transactions on Serious Games EAI for Innovation 7 04 2013 – 05 2014 | Volume 1 | Issue 2 | e5 H. van Oostendorp et al. [19] Niehaus, J. and Riedl, M.O. (2009) Scenario [32] Van der Spek, E. D. (2011) Experiments in serious adaptation: an approach to customizing computer- game design: a cognitive approach. SIKS Dissertation based training games and simulation. In Proceedings Series, Utrecht University, The Netherlands. AIED Workshop Intell Edu. Game (Brighton, UK), 89- [33] Ferguson, R. (2012) Learning analytics: drivers, 98. developments and challenges. International Journal of [20] Lavie, T., & Meyer, J. (2010) Benefits and costs of Technology Enhanced Learning, 4(5/6): 304-317. adaptive user interfaces. International Journal of [34] PCKNOT [Computer Software]. Las Cruces, NM: Human-Computer Studies, 68, 508-524. Interlink Inc. [21] Rocksmith [Computer software]. San Francisco, CA: [35] Schvaneveldt, R. W., Durso, F. T. and Dearholt, D. W. Ubisoft. (1985) Pathfinder: Scaling with network structures [22] Mario Kart [Computer software]. Kyoto, Japan: (Memorandum in Computer and Cognitive Science, Nintendo. MCCS-85-9, Computing Research Laboratory). Las [23] Need for Speed [Computer software]. Redwood City, Cruses: New Mexico State University. CA: Electronic Arts. [36] Gomez, R. L., Hadfield, O. D. and Housner, L. D. [24] Yun, C., Shastri, D., Pavlidis, I. and Deng, Z. (2009) (1996) Conceptual maps and simulated teaching O' game, can you feel my frustration?: Improving episodes as indicators of competence in teaching user's gaming experience via stresscam. In elementary mathematics. Journal of Educational Proceedings of the 27th international conference on Psychology, 88(3): 572-585. Human factors in computing systems (CHI '09) (New [37] Day, E. A., Arthur Jr., W. and Gettman, D. (2001) York: ACM), 2195-2204. Structural knowledge assessment: Comparison of [25] The Elder Scrolls 4: Oblivion [Computer software]. referent structures. Journal of Educational Psychology, Rockville, MD: Bethesda. 86(5): 1022-1033. [26] Shute, V. J., Ventura, M., Bauer, M. I. and Zapata- [38] Wouters, P.J.M., van der Spek, E.D. and van Rivera, D. (2009) Melding the power of serious games Oostendorp, H. (2011) Measuring learning in serious and embedded assessment to monitor and foster games: a case study with structural assessment. learning: Flow and grow. In U. Ritterfeld, M. Cody, & Educational technology research and development, P. Vorderer (eds.), Serious games: Mechanisms and 59(6): 741-763. effects (New York, NY: Routledge), 295-321. [39] Lessiter, J., Freeman, J., Keogh, E. and Davidoff, J. [27] Bostan, B. and Ogut, S. (2009) Game challenges and (2001) A cross-media presence questionnaire: The difficulty levels: lessons learned From RPGs. In ITC-sense of presence inventory. Presence: International Simulation and Gaming Association Teleoperators and Virtual Environments, 10 (3): 282- Conference. Retrieved from 297. http://www.silentblade.com/presentations/Bostan_Ogu [40] Graesser, A.C., Chipman, P., Leeming, F. and t_Full_Paper.pdf Biedenbach, S. (2009) Deep learning and emotion in [28] Bailey, C. and Katchabaw, M. (2005) An experimental serious games. In U. Ritterfeld, M. Cody & P. testbed to enable auto-dynamical difficulty in modern Vorderer (eds.), Serious Games: Mechanisms and video games. In Proceedings of the 2005 GameOn Effects (New York: Routledge), 81-100. North America Conference, 18-22. [41] Erhel, S. and Jamet, E. (2013) Digital game-based [29] Half-Life 2 [Computer software]. Bellevue, WA: learning: impact of instructions and feedback on Valve Corporation. motivation and learning. Computers & Education, 67: [30] Van der Spek, E. D., Wouters, P. and van Oostendorp, 156-167. H. 2011) Code Red: Triage Or COgnition-based [42] Butler, A.C., Godbole, N. and Marsh, E.J. (2013) DEsign Rules Enhancing Decisionmaking TRaining In Explanation Feedback is Better Than Correct Answer A Game Environment. British Journal of Educational Feedback for Promoting Transfer of Learning. Journal Technology, 42(3): 441-455. of Educational Psychology, 105(2): 290-298. [31] Van der Spek, E.D., van Oostendorp, H. and Meyer, J- J.Ch. (2013) Introducing surprising events can stimulate deep learning in a serious game. British Journal of Educational Technology, 44(1): 156-169. European Alliance EAI Endorsed Transactions on Serious Games EAI for Innovation 04 2013 - 05 2014 | Volume 1 | Issue 2 | e5 http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png EAI Endorsed Transactions on Game-Based Learning Unpaywall

Adapting the Complexity Level of a Serious Game to the Proficiency of Players

EAI Endorsed Transactions on Game-Based LearningMay 22, 2014

Loading next page...
 
/lp/unpaywall/adapting-the-complexity-level-of-a-serious-game-to-the-proficiency-of-qMd02nmxnb

References

References for this paper are not available at this time. We will be adding them shortly, thank you for your patience.

Publisher
Unpaywall
ISSN
2034-8800
DOI
10.4108/sg.1.2.e5
Publisher site
See Article on Publisher Site

Abstract

As games are continuously assessing the player, this assessment can be used to adapt the complexity of a game to the proficiency of the player in real time. We performed an experiment to examine the role of dynamic adaptation. In one condition, participants played a version of our serious game for triage training that automatically adapted the complexity level of the presented cases to how well the participant scored previously. Participants in the control condition played a version of the game with no adaptation. The adapted version was significantly more efficient and resulted in higher learning gains per instructional case, but did not lead to a difference in engagement. Adapting games to the proficiency of the player could make serious games more efficient learning tools. Keywords: dynamic adaptation, engagement, learning efficiency, proficiency, serious game Received on 29 December 2013, accepted on 05 March 2014, published on 22 May 2014 Copyright © 2014 H. van Oostendorp et al., licensed to ICST. This is an open access article distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/3.0/), which permits unlimited use, distribution and reproduction in any medium so long as the original work is properly cited. doi: 10.4108/sg.1.2.e5 [4], and some learners will therefore benefit from a slower 1. Introduction pace in the presentation of instructional material in order to correctly organize all the new information that is coming in. Serious games can be used to engender learning in a player, Conversely, efficient learning may also be hindered by and two recent meta-analyses have shown that the usage of cognitive underload, where the learner is stimulated too serious games may even lead to superior learning compared little, for instance when a quick learner plays a game that with traditional (but passive) instructional methods [1, 2]. has a slow pace in order to accommodate slow learners. However a serious game is found to be primarily efficacious Cognitive underload can lead to (passive) fatigue, which has if a person is allowed to play the game multiple times [1], a been shown to result in disengagement from the task and result that Wouters and Van Oostendorp [3] argue higher distractibility and can subsequently degrade underlines the notion that games are complex environments performance [5, 6]. If a game were to actively prevent the in which the player first has to learn how to control the player from becoming cognitively overloaded or game and the way in which it conveys the instructional underloaded, it could therefore be more efficient [7]. material, before this material itself can be learned. Games in Secondly and closely related to this, Csikszentmihalyi turn are products that have to be made beforehand and have [8] posited that one can experience the feeling of flow, a preset pace, and often do not take into account the which is a feeling where someone is completely engaged in individual learning rate. an activity to the point of losing self-consciousness and the People learn at different speeds, which may lead to a activity becomes rewarding in its own right, and that this number of problems. Firstly, the rich multimodal leads to the individual functioning at his or her fullest information of a game may overload the limited working capacity [9]. This is achieved when the provided challenge memory capacity of a player, leading to incorrect learning is optimally suited to the skills of the user; and as ___________________________________ videogames are often stated to be engaging, with players Corresponding author : h.vanoostendorp@uu.nl European Alliance EAI Endorsed Transactions on Serious Games EAI for Innovation 04 2013 - 05 2014 | Volume 1 | Issue 2 | e5 H. van Oostendorp et al. reporting an experience of being completely absorbed in the (serious) games are developed, and progress has been made, game, they seem to be ideally suited to produce flow [10, empirical research to effects of adaptivity in terms of 11]. Flow has been shown to be positively correlated to learning and engagement are still scarce [see also 20].. In learning [12]; therefore, keeping players in a sense of flow this paper we will remedy this and present results of an by adjusting the challenge to their skills could improve empirical study on the learning and affective effects of a learning [13]. game with dynamic adaptivity. That is, a game where the Summarizing, if quick learners were able to progress in challenges of, or difficulties caused by, the game are the game at a faster pace, for instance because the game increasing, and at a rate dependent on the proficiency of the recognizes their proficiency and adapts the game player (online adaptation). We will mainly be concerned accordingly, engagement in performing the task could be with varying the attributes of the non-player characters. enhanced which in turn results in a higher efficiency of the game. Similarly, a slower pace for slower learners would 2.2. Assessing the proficiency of players also improve engagement and efficiency for them. In this paper we will examine in an experimental study whether For the principle of fitting the instruction to the learner's adapting a serious game to the proficiency of players proficiency level to be implemented in serious games improves learning and engagement. But first we will, in the effectively, it is important first that the proficiency should next section, discuss different aspects of adaptivity in be assessed and secondly that the challenge should be general, how we monitored or assessed proficiency of adapted to the player automatically in a non-obtrusive way. players and how we implemented adaptivity in a dynamic Automatically assessing and adapting the challenge or way in the serious game Code Red Triage. difficulty of a game to the proficiency of a player is slowly becoming commonplace in entertainment games. For instance in Rocksmith [21], a musical instrument simulation 2. Adaptivity game, the player needs to hit the correct notes of a song with good timing. The game adds more notes and places a greater 2.1. Aspects of adaptivity emphasis on timing when the player performs well, or vice versa when the player performs badly. Racing games like In line with Lopes and Bidarra [14], we can distinguish Mario Kart [22] and Need for Speed [23], implement a several components of adaptation. 1) The game world and simple adaptation known as ‘rubber banding’: when the its objects can be varied, e.g. the layout of the game world player lags behind the other racing contestants, they will can be made simpler for underachieving players [15]. 2) The slow down in order to let the player catch up with them – game play mechanics, how game elements work, including when the player is up front, his opponents will become actions like running or shooting, e.g. adjusting shooting faster and try to keep up with him. difficulty by providing player aim assistance, according to Here, we will elaborate on two modes of assessing that individual skills [16]. 3) Adapting the attributes of the non- are most relevant to our research. Firstly, one interesting player characters in the game, e.g. increasing the abilities of avenue in which a game can be adapted to the player was the non-playing character when the player performs well. undertaken by Yun et al. [24], who used an infrared camera Domain knowledge is here automatically gathered by the that was mounted on a TV displaying the game. This camera game based on Artificial Intelligence-techniques, in order to (overtly) recorded the faces of the participants while they offer more challenging behavior of the non-player characters were playing a game that revolved around shooting enemy [17]. 4) Game narratives, e.g. adapting the sequence of robots. Looking at the heat signatures from the supra-orbital events to the pace or behavior of the player [18], and 5) region of the face, they were able to derive how much game scenarios - more or less similar to the previous one: apparent stress the game exerted on the player during game adapting the flow of events and actions within a game, that play. At the same time, the player reported at set intervals is, adapting the progression within a game level to the whether they found the game too easy, just right or too learning goals of the player. For instance, monitoring the difficult, and whether they were enjoying the game or would players actions and based on that certain points in the plot like to quit. This research is relevant to our own for two are included in the game (or not) [19]. reasons. One, they discovered that people who found the A next issue in creating adaptive games is to decide on game too difficult and wanted to quit actually had lower the method of generating the content. Lopes and Bidarra stress levels than when the game was moderately difficult. distinguish two general methods. First, offline adaptivity (or They argued that this is due to the player becoming customized content generation); adjustments are made disengaged with the game, thereby corroborating the considering player-dependent data, but prior to initiating the previously made assertion that too high a challenge leads to gameplay. Secondly, online adaptivity, i.e. adjusting the cognitive overload and is detrimental to the engagement or game to its players, in real time, as they play. flow experience. Two, a version of the game where the A further discussion on the way adaptation can be game automatically assesses and adapts to the stress level of implemented in games and the associated challenges can, for the player was shown to lead to higher engagement and instance, be found in Lopes and Bidarra [14]. Though in the better in-game performance (in terms of how many robots (game) industry and academia now many different adaptive European Alliance EAI Endorsed Transactions on Serious Games EAI for Innovation 04 2013 - 05 2014 | Volume 1 | Issue 2 | e5 Adapting the Complexity Level of a Serious Game to the Proficiency of Players were defeated) than in conditions with preset difficulty attention. The mobility (sieve) triage taught here is a levels, even for the easy difficulty level. relatively simple procedure, where it takes the first Another interesting example of how to adapt the game to responder between one and five steps to determine the the player is the entertainment game The Elder Scrolls 4: severity of the victim’s injuries. When the game starts, the Oblivion [25]. Here, the player roleplays a character in a player finds himself in an empty train station with signs of large and open medieval fantasy world. As the player recent panic. Here, he learns that he is a medical first encounters new locales, performs quests and defeats responder who has received a call that a bomb has gone off monsters, his or her character will gradually become on a subway platform. The player is then told to find the stronger and gain better weapons and items (see further subway platform and perform the triage procedure on the Shute et al., [26]). Because the game features an open world victims. Upon reaching the subway platform (see Figure 1), for the player to explore freely, this traditionally leads to a visible timer starts counting down from seventeen minutes. problems where the player may encounter monsters that are When the timer reaches zero, the game ends. This timer was far too strong for his or her avatar to defeat at that point in added to instill a sense of immediacy and stress; in practice time. To counter this and provide the optimal experience for almost every participant is able to triage all victims everyone, the player’s adversaries in the game also progress comfortably within this time. At the subway platform, the in power at the same rate as the skill level of the player. player can then walk up to a victim and press a button to Contrarily to what would be expected, many gamers enter the triage menu, which consists of eight buttons for criticized this feature, as it made them feel that their actions triage actions, and four buttons for the four different triage were largely inconsequential [27]; they were not getting categories (see Figure 2). Pressing a triage button will give a stronger than their enemies and therefore they didn’t feel few lines of general information on what the action entails like they were mastering the game. and approximately at what stage in the procedure it should Above we mentioned two different techniques of be used, and a line with specific information on how the assessing the player proficiency within the game. The first action affected the victim the player’s looking at. After was a more overt technique, where in real life settings the choosing a few triage actions the player should be able to player would have to install an infrared camera for it to have an idea how heavily injured the victim is and assign a work; the second example featured so-called ‘stealth’ triage category. assessment [7, 26], that is, a more covert assessment that is coupled to the naturally occurring moves of the player in the game. In essence, all games are an assessment device, in that progressing past an obstacle is contingent on acquiring the needed knowledge of how to do so. As digital games are played on computers, which require that every game rule and in-game problem encountered is computable, determining whether the player succeeded is often easily quantifiable. 2.3. Dynamic adaptivity in the serious game Code Red Triage As indicated we want to study whether the online adaptation of the challenge or difficulty of a learning experience to the proficiency of players, improves learning and enhances engagement. Following [28] we use the term dynamic adaptivity to designate online adaptation of game experiences in terms of complexity and matching that to the proficiency of players. In order to test this hypothesis we used the serious game Code Red Triage, a total conversion mod of Half-Life 2 [29-31]. The game is designed to teach Figure 1. Subway platform in the game Code Red the triage procedure, a procedure for medical first Triage responders to prioritize the victims of a mass casualty event according to how urgently the victim needs medical European Alliance EAI Endorsed Transactions on Serious Games EAI for Innovation 3 04 2013 – 05 2014 | Volume 1 | Issue 2 | e5 H. van Oostendorp et al. Figure 2. Triage menu in the game Figure 3. Feedback after categorizing a victim in a triage category Once this is done, the victim changes color to depict the operationalized as the game deleting all remaining victim chosen category and the player receives a score showing cases within the same tier, if the player scored higher than how well he did, as well as a few lines telling him whether a threshold value for that victim. The threshold was or not a) he forgot to take procedure steps, b) took steps in determined with the data from a pilot experiment, by the wrong order, c) took unnecessary steps and d) whether rounding up the average score per victim tier. A player it was done within the allotted time (between 10 and 55 who was unable to triage a victim case and scored below seconds), see Figure 3 for a screenshot. The in-game score the threshold, received one or more of the remaining cases that can be obtained per victim ranges from 0 to 100 and of that tier before going to the next level of complexity. In is based on the previous four criteria. other words, more successful players could attain the most In the case of Code Red Triage, we already have a complex case in less cases, and consequently learn to measure to assess how well the player is performing in the perform the triage more efficiently. In the control version game, namely the in-game score, which provides us with of the game all (19) cases were presented in a gradually an objective measure of whether the player is able to increasing complexity. correctly apply the procedure to a given victim case. The We hypothesize that players feel more engaged by the player’s performance can therefore be seen as an dynamic adaptive version, because the game always indication of their proficiency level [7]. We can thus use remains challenging (compared to a control version), and the above mentioned covert method to assess the secondly we expect in the dynamic adaptive version of the proficiency of players here. game that players are able to learn more efficiently, We used this in-game score to adapt the difficulty of because redundant learning experiences (triage cases) can the game to the proficiency of the player. In Code Red be skipped. Triage, there are a total of six paths with an increasing number of steps in the triage procedure that are taught 3. Method with the game, but there are multiple victims for any given path. As the victims are encountered in increasing order of complexity (i.e. the number of steps needed to 3.1. Participants come to a correct categorization), these groups of victims are called ‘victim tiers’. In the set of victims 6 tiers or In total 28 individuals of university-level education, 19 levels of complexity were distinguished. In other words, male and 9 female, participated in the experiment, and the attributes of the non-player characters were varied in were randomly assigned to the adaptive game condition complexity. If a player scores above a preset threshold, he (n=14), and the control condition (n=14). Average age or she has proven to have a certain level of proficiency was 22.86 with a standard deviation of 5.68. and can move on to a more complex victim tier. In the adaptive condition of Code Red Triage this was European Alliance EAI Endorsed Transactions on Serious Games EAI for Innovation 04 2013 - 05 2014 | Volume 1 | Issue 2 | e5 Adapting the Complexity Level of a Serious Game to the Proficiency of Players 3.2. Materials 3.3. Apparatus and procedure To measure the learning of players, three types of instruments were used. The in-game score (see above) The game was played on a 17” laptop at a resolution of formed the first measure: an indication of the progression 1920 x 1200 with circum-aural headphones in a room with of the player in the game. In several studies done with the the lights turned off. The graphics settings were set at same game and the same in-game score we found that the their maximum and the game ran at a constant 60 frames in-game score significantly correlated with a knowledge per second. The participants were asked to perform the test presented after the game [31, 32], which gives structural knowledge assessment with the PCKNOT plausibility to the notion that the in-game score, conceived software. Then, the knowledge test was administered. as analytical learning tool [33], is a valid measure of Before playing the game, the participants were given learning. Statistics from the game that were logged instructions about Code Red Triage and were informed furthermore included triaged victims, number of triaged about its goal. Nothing was revealed to them about the victims, tier of victim, time per victim, total time, score condition they took part in. Playing the game from start to per victim and total score. Second and third, we measured finish took each participant at most 25 minutes: a few how much a participant learned in the game with two minutes for the entry level, a few more for the hallway measures: a pen-and-paper knowledge test and a structural part and a maximum of 17 minutes was allowed for the knowledge assessment. The knowledge test was in the metro platform part, in which the triages took place. The form of eight verbal and eight pictorial multiple choice scores participants reached in the game gave information questions where the player had to answer questions related about their performance (see also section 2.3). Directly to the triage procedure by choosing one of four after the participants finished playing the game, they were alternatives (total score range 0-16). asked to fill out the engagement questionnaire. They were Whereas the knowledge test measured how well the then asked to do the structural knowledge assessment and participant could reproduce declarative knowledge, the knowledge test as before, but with the questions in a structural knowledge assessment determined how the different order. Finally, the participants were thanked for information was organized on a deeper, more structural their cooperation and they received a coupon for their level. Here, a computer program called PCKNOT [34] was work. An overview of the procedure can be seen in Figure used, that let participants rate the degree of relatedness of pairs of concepts from the triage procedure. These ratings could subsequently be used to elicit a participant’s knowledge structure with the Pathfinder metric [35] and compared to the knowledge structure of experts; resulting in a similarity measure that indicated how well the participant had organized the information of the triage Figure 4. Procedure of the experiment procedure structurally [36]. The score range varies from - 1 through 0 to +1. Pathfinder has been successfully applied by [37] to measure learning from a complex videogame. They found that it was also predictive of skill 4. Results retention and skill transfer. For further information see Engagement Wouters, Van der Spek and Van Oostendorp [38]. In our case we focused on 8 important concepts from the triage The mean scores and standard deviations of the procedure and consequently 28 pairs were presented for engagement questionnaire are mentioned in Table 1. An the related judgments. The created networks were ANOVA showed no significant effect of condition on the compared with the referent structure that was derived by ITC-SOPI engagement questionnaire, F(1,26) < 1. averaging the elicited knowledge structures of the current researchers. Learning Efficiency The engagement of players was measured by using the subscale of the ITC Sense of Presence Inventory (ITC- There are several ways to determine whether learning was SOPI), which indicates the participant’s feelings of more efficient in the adaptive condition. A reliable engagement with a twelve item five-point Likert scale measure for efficiency is to divide the posttest scores of [39]. If the challenge of the game is better adjusted to the the participants by the number of victim cases triaged, abilities of the player, one would expect the player to be giving us an indication of how much the participant has drawn into the game more, which we hoped to see learned per unit of instruction, and whether this would be expressed in the scores on this subscale. The reliability of higher in a game that adapts the information presentation the ITC-SOPI Engagement questionnaire appeared to be to the player’s proficiency. Another way would be to relatively low, Cronbach's coefficient α = 0.59. divide learning performance by total time spent playing the game. However some players navigate more efficiently than others towards the platforms etc, which European Alliance EAI Endorsed Transactions on Serious Games EAI for Innovation 5 04 2013 – 05 2014 | Volume 1 | Issue 2 | e5 H. van Oostendorp et al. blurs what we want to measure. We therefore decided to point. The intervention itself may be too small next to all use learning performance divided by the number of cases the other determinants of engagement, such as the game’s triaged, as a purer measure of learning efficiency. setting, world, expectations, control interface, et cetera, to show up as a difference on the rating scale, but the adaptive version may still be preferred when the conditions were placed side by side. A second explanation could be related to the fact that we only asked participants Table 1: Mean engagement and efficiency scores on to appraise their engagement after the game. It is unclear knowledge test and structural knowledge assessment whether a continuous measurement of a participant’s (sd). engagement, for instance with an infrared camera as in the research by [24], as we mentioned in the introduction, would have resulted in higher ratings throughout the game Control Adaptive in the adaptive version. Thirdly, people may play games Condition Condition for different reasons; a higher challenge could lead to Engagement (1-5) 3.63 (.33) 3.66 (.45) higher engagement in some players, whereas it has the opposite effect on others. Lastly, and perhaps as a result Knowledge test .57 (.19) 1.02 (.30) of the previous explanation, we found that the Structural homogeneity of the engagement questionnaire knowledge .015 (.004) .028 (.019) (Cronbach’s alpha) was low. Perhaps this measurement assessment problem contributed to the fact that we did not find an effect of engagement. An ANCOVA with the pretest as covariate, condition as We saw that participants learned more per victim case fixed factor and posttest score divided by the total number in the adaptive condition compared to the control of victims triaged as dependent variable showed that condition. It could be that the moment a participant grasps condition had a significant effect on both the knowledge the procedure to resolve a victim case pertaining to a test (F(1,25) = 21.98, p < .001, d = 1.81) and the structural certain tier, the information presented in the following knowledge assessment (F(1,25) = 5.05, p < .05, d = .89). victims in that tier is redundant, at least to a point that it The means on these relative measures and standard does not improve learning of the procedure anymore, deviations of these tests are listed in Table 1. making the adaptive version more efficient. In order to determine whether the adaptive condition In-game score not only made learning the instructional material more efficient, but also leads to deeper learning [40], other The total in-game score was significantly higher for the experiments should be set up such as e.g. a study where control condition (M = 777.7, SD = 321.2) than for the learning is also measured after a longer delay or with adaptive condition (M = 316.4, SD = 107.8), F(1,26) = transfer tasks. However, some corroboration may be 25.95, p < .001, however this more or less follows from found in the structural knowledge assessments. They point the result that participants triaged significantly less to deeper learning in the adaptive condition. victims in the adaptive condition. One last observation concerns the relation between engagement and learning; the results found indicate that an increase in engagement does not seem necessary to 5. Conclusion and discussion enhance learning efficiency. Also the correlation between engagement and learning efficiency appeared to be low We hypothesized that a serious game that dynamically and not significant (p > .05) for both groups of adapts its challenge, or complexity presentation, to quick participants. However, for this finding too, the same learners could make a serious game more engaging and remarks as before should be made concerning the more efficient. The first part of the hypothesis was not measured engagement of players. confirmed, while the second part was confirmed; All in all, a rather simple alteration of a serious game participants in the adaptive game version learned where it dynamically adapts the presentation of significantly more per victim case than in the control complexity to the player’s performance and thereby its condition, and were therefore more efficient. challenge has been shown to markedly improve the We found no difference in the engagement ratings. If efficiency thereof. This is a promising result for serious the improved learning per unit of instruction was due to games developers that worry about the comparative less disengagement from the task, one would expect this efficiency of their game, as well as for researchers to appear from the results of the engagement interested in improving games with the aid of more questionnaire. We propose four explanations why we did sophisticated adaptation engines. It can also be a useful not find a difference in engagement. result for entertainment game developers, as many games Firstly, when participants had to appraise their need to incorporate tutorial levels that are necessary for engagement just after playing the game, they lacked players to understand the game, but are not a lot of fun to knowledge of the other condition and thereby a reference play, especially upon repeated playthroughs. A dynamic European Alliance EAI Endorsed Transactions on Serious Games EAI for Innovation 04 2013 - 05 2014 | Volume 1 | Issue 2 | e5 Adapting the Complexity Level of a Serious Game to the Proficiency of Players [3] Wouters, P.J.M., and van Oostendorp, H. (2013) A adaptive version that adapts to the player's proficiency meta-analytic review of the role of instructional could greatly speed up these mandatory instructional support in game-based learning. Computers & sequences and (possibly) make them more challenging. Education, 60: 412-425. [4] Moreno, R. and Mayer, R. (2007) Interactive Future research multimodal learning environments. Educational Above we already mentioned two limitations to our study, Psychology Review, 19(3): 309-326. viz. that it is impossible to conclusively state whether [5] Saxby, D. J., Matthews, G., Hitchcock, E. and Warm, dynamically adapting to the player’s performance only J. S. (2007) Development of active and passive fatigue manipulations using a driving simulator. In resulted in more efficient instruction, or also in deeper Proceedings of the Human Factors and Ergonomics learning, and that it is unclear whether participants Society 51st Annual Meeting (Santa Monica, CA: differed in engagement during gameplay. In addition, Human Factors and Ergonomics Society), 1237-1241. another limitation of our experimental setup that warrants [6] Paas, F., Renkl, A. and Sweller, J. (2004) Cognitive future research is that we did not measure retention over load theory: Instructional implications of the longer time periods. Participants in the adaptive game interaction between information structures and version received less practice and consequently less cognitive architecture. Instructional Science, 32(1): 1- opportunity to internalize the information. Therefore there is a real possibility - or even danger - that the participants [7] Shute, V. J. (2011) Stealth assessment in computer- based games to support learning. In S. Tobias & J.D. in the dynamic adaptive condition remember less of the Fletcher (eds.), Computer games and instruction instruction after several weeks. Regarding dynamic (Charlotte, NC: Information Age), 503-524. adaptation itself, in this study we did make some specific [8] Csikszentmihalyi, M. (1988) The flow experience and choices during the design and implementation process. its significance for human psychology. In M. We focused on the nature of the non-player characters and Csikszentmihalyi & I.S. Csikszentmihalyi (eds.), let them vary in number of steps needed to perform a Optimal experience: Psychological studies of flow in correct triage. Several alternatives are open for continued consciousness (Cambridge, MA: Cambridge research to the role of dynamic adaptation. For instance, University Press), 15-35. the set of buttons for executing the triage actions could be [9] Shernoff, D. J., Csikszentmihalyi, M., Shneider, B. and Shernoff, E. S. (2003) Student engagement in high adapted, that is, starting simple and increasing over time, school classrooms from the perspective of flow theory. depending on performance. Or the feedback given to School Psychology Quarterly, 18 (2): 158-176. players could be adapted, e.g. stating more or less [10] Sweetser P. and Wyeth, P. (2005) GameFlow: a model explicitly what went right and what went wrong while for evaluating player enjoyment in games. Computers performing the triage [41], or explanatory feedback could in Entertainment, 3(3): 1-24. be included, particularly in early learning phases [42]. [11] Cowley, B., Charles, D., Black, M. and Hickey, R. Finally, another option for making training procedures (2008) Toward an understanding of flow in video adapt themselves to participants is the notion of games. Computers in Entertainment 6(2): 1-27. adaptability. In this form of offline adaptation participants [12] Webster, J., Trevino, L. K. and Ryan, L. (1993) The dimensionality and correlates of flow in human- indicate themselves what direction they want to practice computer interactions. Computers in Human Behavior, and what part of the procedure they want to repeat. These 9(4): 411-426. are questions that still need to be examined in the future. [13] Liu, M., Horton, L., Olmanson, J. and Toprac, P. (2011) A study of learning and motivation in a new media enriched environment for middle school Acknowledgements science. Education Technology Research Development, 59: 249-265. This research has been supported by the GATE project [14] Lopes, R. and Bidarra, R. (2011) Adaptivity (http://gate.gameresearch.nl/), funded by the Netherlands challenges in games and simulations: a survey. IEEE Organization for Scientific Research (NWO). Transactions on Computational Intelligence and AI in Games 3(2): 85-99. [15] Walker, J. (2009) Left 4 Dead 2: Exclusive RPS Hand on Preview. Rock Paper Shotgun, PCGaming since References 1873. Online available: http://www.rockpapershotgun.com/2009/06/01left-4- [1] Sitzmann, T. (2011) A meta-analytic examination of dead-2-exclusive-rps-preview. the instructional effectiveness of computer-based [16] Brathwaite, B. and Schreiber, I. (2009) Challenges for simulation games. Personnel Psychology: 489-528. game designers, (Boston, MA: Charles River Media). [2] Wouters, P.J.M., van Nimwegen, C., van Oostendorp, [17] Bakkes, S., Spronck, P. and van den Herik, J. (2009) H. and van der Spek, E.D. (2013) A Meta-analysis of Rapid and reliable adaptation of videogame AI. IEEE the Cognitive and Motivational Effects of Serious Transactions Computational Intelligence and AI in Games. Journal of Educational Psychology, 105: 249- Games, 1(2): 93-104. [18] Roberts, D.L. and Isbell, C.L. (2008) A survey and qualitative analysis of recent advances in drama management. Int. Trans. Sci. Appli., 3(2): 61-75. European Alliance EAI Endorsed Transactions on Serious Games EAI for Innovation 7 04 2013 – 05 2014 | Volume 1 | Issue 2 | e5 H. van Oostendorp et al. [19] Niehaus, J. and Riedl, M.O. (2009) Scenario [32] Van der Spek, E. D. (2011) Experiments in serious adaptation: an approach to customizing computer- game design: a cognitive approach. SIKS Dissertation based training games and simulation. In Proceedings Series, Utrecht University, The Netherlands. AIED Workshop Intell Edu. Game (Brighton, UK), 89- [33] Ferguson, R. (2012) Learning analytics: drivers, 98. developments and challenges. International Journal of [20] Lavie, T., & Meyer, J. (2010) Benefits and costs of Technology Enhanced Learning, 4(5/6): 304-317. adaptive user interfaces. International Journal of [34] PCKNOT [Computer Software]. Las Cruces, NM: Human-Computer Studies, 68, 508-524. Interlink Inc. [21] Rocksmith [Computer software]. San Francisco, CA: [35] Schvaneveldt, R. W., Durso, F. T. and Dearholt, D. W. Ubisoft. (1985) Pathfinder: Scaling with network structures [22] Mario Kart [Computer software]. Kyoto, Japan: (Memorandum in Computer and Cognitive Science, Nintendo. MCCS-85-9, Computing Research Laboratory). Las [23] Need for Speed [Computer software]. Redwood City, Cruses: New Mexico State University. CA: Electronic Arts. [36] Gomez, R. L., Hadfield, O. D. and Housner, L. D. [24] Yun, C., Shastri, D., Pavlidis, I. and Deng, Z. (2009) (1996) Conceptual maps and simulated teaching O' game, can you feel my frustration?: Improving episodes as indicators of competence in teaching user's gaming experience via stresscam. In elementary mathematics. Journal of Educational Proceedings of the 27th international conference on Psychology, 88(3): 572-585. Human factors in computing systems (CHI '09) (New [37] Day, E. A., Arthur Jr., W. and Gettman, D. (2001) York: ACM), 2195-2204. Structural knowledge assessment: Comparison of [25] The Elder Scrolls 4: Oblivion [Computer software]. referent structures. Journal of Educational Psychology, Rockville, MD: Bethesda. 86(5): 1022-1033. [26] Shute, V. J., Ventura, M., Bauer, M. I. and Zapata- [38] Wouters, P.J.M., van der Spek, E.D. and van Rivera, D. (2009) Melding the power of serious games Oostendorp, H. (2011) Measuring learning in serious and embedded assessment to monitor and foster games: a case study with structural assessment. learning: Flow and grow. In U. Ritterfeld, M. Cody, & Educational technology research and development, P. Vorderer (eds.), Serious games: Mechanisms and 59(6): 741-763. effects (New York, NY: Routledge), 295-321. [39] Lessiter, J., Freeman, J., Keogh, E. and Davidoff, J. [27] Bostan, B. and Ogut, S. (2009) Game challenges and (2001) A cross-media presence questionnaire: The difficulty levels: lessons learned From RPGs. In ITC-sense of presence inventory. Presence: International Simulation and Gaming Association Teleoperators and Virtual Environments, 10 (3): 282- Conference. Retrieved from 297. http://www.silentblade.com/presentations/Bostan_Ogu [40] Graesser, A.C., Chipman, P., Leeming, F. and t_Full_Paper.pdf Biedenbach, S. (2009) Deep learning and emotion in [28] Bailey, C. and Katchabaw, M. (2005) An experimental serious games. In U. Ritterfeld, M. Cody & P. testbed to enable auto-dynamical difficulty in modern Vorderer (eds.), Serious Games: Mechanisms and video games. In Proceedings of the 2005 GameOn Effects (New York: Routledge), 81-100. North America Conference, 18-22. [41] Erhel, S. and Jamet, E. (2013) Digital game-based [29] Half-Life 2 [Computer software]. Bellevue, WA: learning: impact of instructions and feedback on Valve Corporation. motivation and learning. Computers & Education, 67: [30] Van der Spek, E. D., Wouters, P. and van Oostendorp, 156-167. H. 2011) Code Red: Triage Or COgnition-based [42] Butler, A.C., Godbole, N. and Marsh, E.J. (2013) DEsign Rules Enhancing Decisionmaking TRaining In Explanation Feedback is Better Than Correct Answer A Game Environment. British Journal of Educational Feedback for Promoting Transfer of Learning. Journal Technology, 42(3): 441-455. of Educational Psychology, 105(2): 290-298. [31] Van der Spek, E.D., van Oostendorp, H. and Meyer, J- J.Ch. (2013) Introducing surprising events can stimulate deep learning in a serious game. British Journal of Educational Technology, 44(1): 156-169. European Alliance EAI Endorsed Transactions on Serious Games EAI for Innovation 04 2013 - 05 2014 | Volume 1 | Issue 2 | e5

Journal

EAI Endorsed Transactions on Game-Based LearningUnpaywall

Published: May 22, 2014

There are no references for this article.