ADL Newsletter for Educators and Educational Researchers

Training for Adaptable Performance: A Workshop Report

Dick Clark
Center for Cognitive Technology
University of Southern California

Introduction

This report summarizes the result of a two-day "Adaptability Research Workshop" sponsored by the Institute for Creative Technologies and the Center for Cognitive Technology at the University of Southern California. Participants included 15 military and contractor researchers who have current responsibilities for directing or planning training R&D programs that are attempting to increase adaptable performance. Also invited were 10 university researchers who specialize in the design of instructional experiments emphasizing adaptability. An effort was made to invite prominent researchers who disagreed with each other about adaptability strategies, theories and research designs.

Background of the Workshop

Two parallel events have recently pushed adaptability to the forefront of both military and academic research and development goals. Department of Defense (DoD) planners have requested (see for example, Burns and Freeman, 2008) recommendations for changes in training methods that would result in more adaptable military performance in future asymmetric warfare situations, where an organized force has to deal with insurgencies or guerilla forces such as in Iraq and Afghanistan. The Burns and Freeman (2008) report summarized a number of promising efforts to achieve more adaptable performance after training but complained about a lack of solid research and evaluation in many programs. It recommended a variety of demonstration projects and experiments to support improvements and specifically requested the involvement of "behavioral and social scientists with experience in adaptability-related training and education" (p. S-3, Burns and Freeman, 2008).

A second stimulus for the conference stems from a recent dialogue among instructional researchers in a number of nations about the most effective instructional methods for teaching any subject or task. The most recent marker in this discussion is a book edited by Tobias and Duffy (2009) where various authors argue for different instructional approaches. Recent publications, including a research review authored by Kirschner, Sweller and Clark (2006), have challenged the dominant constructivist paradigm for training (AKA problem-based learning, discovery learning, inquiry learning) and claimed that the past half-century of research provides overwhelming support for an alternative approach often called "direct instruction". Reactions to the argument were many and varied in the Tobias and Duffy, 2009 volume. One criticism of direct instruction raised in the ensuing debate was that while it might often result in more learning, various forms of direct instruction might also inhibit adaptable performance.

Many of the academic researchers involved in the debate have complained that a lack of collaborative research has limited the usefulness of experiments that tested various hypotheses focused on these issues. Researchers tend to collaborate and communicate more frequently with other researchers who prefer similar theories and research methods. In a typical study the learning and transfer (adaptability) benefits of two or more instructional approaches are compared in studies where learners, learning tasks and outcome measures are held constant. Yet since researchers who are committed to one point of view design most of these studies they tend to unintentionally hobble the competition by setting up a weak "straw man" version of the instructional approach they do not favor. The predictable and questionable result of many of these experiments is that researchers tend to find evidence supporting their own approach.

The Goal of the Workshop

The workshop agenda was designed to begin the process of overcoming the bias implicit in past learning and adaptability research and practice. Of the ten academic researchers invited, half represented a "more constructivist" and half a "more direct instruction" point of view. Military researchers were asked to help their academic colleagues understand the many future challenges of military research, development, training and education.

All participants were asked to collaborate in order to summarize what we know from past research and on the design of future adaptability research and development projects that could be used to support adaptability in cross-cultural encounters and negotiations.

The workshop was organized around three questions:

1) How will we know adaptability when we encounter it?

2) What training and development strategies increase adaptable performance?

3) What research designs are best for investigating the most cost-effective adaptability training methods?

Results of the Discussions:

1. Measures. The first question that was given to the workshop participants was the challenge of assessment – how can we identify adaptive behavior?

Most participants agreed that there is a pressing need for agreement on an operational definition of adaptability, in order to understand what is being evaluated. Current definitions are "many and varied". All participants agreed that adaptability should be assessed using realistic scenarios. Most scenarios should not attempt to capture adaptability as a single construct, but rather, the scenarios should assess different components of adaptability, as identified in a cognitive task analysis. Also, since the outcomes of adaptive behavior depend not only on the actions taken but also on the environment and other factors, adaptability should not be assessed only by the outcomes of actions. Simply put, the same adaptive behavior applied in seemingly similar situations may be successful in one case but fail in another. Instead, the following two aspects could be assessed:

Behavior– do participants demonstrate behavior that is considered adaptive and effective in reaction to unanticipated changes?

Learning – do participants show improvement over time in their reactions to similar changes?

To assess adaptability one should measure improvements in actions, rather than absolute quality of actions (since absolute quality may be the result of prior domain knowledge and not necessarily adaptive behavior). Since adaptability is an ill-defined domain (as there is no predefined "correct" solution to the different scenarios), it seems that there are no automatic tools to assess adaptability. Instead, new measures will have to be developed, and behaviors are likely to be evaluated by human observers. Several techniques were mentioned as possible candidates:

Think aloud protocols

Cognitive Task Analysis

Cued recall

Grading rubrics

Physiological measures (pupil dilation, skin conductivity)

Notably, behavior that is considered adaptive in the short term may be found maladaptive in the long run (and vice versa). Other aspects that were discussed were the unit of analysis (individual, group) and assessing prior domain knowledge.

Nevertheless, participants agreed on several key principles. Mainly, it was agreed that instruction should include many opportunities for practice and feedback using scenarios. Each of these scenarios should be accompanied by feedback in which challenges and correct and incorrect solutions are discussed. With the exception of a possible upfront problem, these scenarios should increase in complexity along several dimensions:

1. From a single skill to multiple skills.
2. From individuals to groups.
3. From routine to more extreme cases and exaggerated reality.
4. From more support and scaffolding (direct instruction) to more independent performance (apprenticeship model).

In order to encourage differentiation, scenarios should differ on deep or structural features such as the scientific and strategic principles that underlie adaptability tactics. In this regard, it was also agreed that trainees should receive some explicit instruction with relevant domain knowledge, and that instruction should also include un-learning, to help participants disrupt their prior learning of dysfunctional responses.

Participants also agreed that additional research would help settle many of the disagreements about teaching methods. Research designs for future studies were identified and will be briefly described. Two researchers representing different theoretical points of view argued that it is impossible to teach adaptability because of the context specificity of knowledge and the lack of evidence for domain-general transfer of learning. The rest of the group agreed that solid evidence supported the use of four methods that increased adaptable learning and performance:

Conduct Task Analysis in advance of instructional design to ensure that what is taught contains accurate and complete descriptions of decision heuristics and performance strategies that are often automated in expert performance.

Provide Conceptual Knowledge Instruction: Teach explicit conceptual knowledge and declarative forms of heuristics related to the tasks and problems students are learning to handle.

Procedural and Problem Solving Instruction: When teaching how to act, analyze or decide, provide realistic scenarios or case studies where adaptive behavior is required. Three types of support during instruction were suggested:

Solved examples: Provide a challenging but representative problem first and after students struggle with it, provide a "how to" demonstration illustrating one or more potential solution strategies drawn from the task analysis of expert performers for them to apply.

Scaffolded problem solving: During hands on application, provide explicit domain-level feedback on their actions. Feedback should include information that is relevant to the current stage of the task and help students identify correct behaviors, and correct mistakes before they are learned.

Varied practice: Provide increasingly novel, unexpected but realistic variations on task relevant scenarios - where possible take them from different knowledge domains.

Reasoning: Require that student's reason about and or explain the previous problem solving process in a way that connects the relevant conceptual knowledge and procedural heuristics employed to perform a task or solve a problem.

Summary: Participants disagreed about the best order for presenting the methods described in 2, 3, and 4 (above) and the relative importance of different methods. All agreed that the methods are most useful for novice to intermediate prior knowledge learners. More advanced learners appear to learn best from scenarios and varied practice alone. Yet there was agreement that we do not now have effective measures of expertise that would allow us to assign trainees to more advanced versions of adaptability instruction and so it may be best to assign everyone to the same instructional conditions.

3. Research plan. The last task given to participants during the workshop was to design a viable research plan to assess these issues. Given the limited time and resources and the wide variety of learning goals, instructional methods, and assessments, workshop participants suggested several possible approaches for instructional design.

Common to all approaches was the need to begin by studying the topic of adaptability, both from the research literature and by conducting thorough cognitive task analyses with highly adaptable leaders. Following that, several participants suggested focusing on non-cognitive issues such as motivation and organizational norms. Later, studies can investigate what instruction improves adaptive behavior. One suggested option was to focus on single elements one at a time (e.g., the addition of complex challenges prior to formal instruction). This can be done in a 2-and-2 design, in which two main forms of instruction are designed (a direct instruction one and a constructivist one). The two instructions are then compared to each other and to variations along single dimensions such as inserting, or avoiding, a problem scenario in advance of training. Another idea was to use established instructional principles to compare several instructional sequences (specifically, a 2x2 design with interleaving solved examples (Y/N); upfront challenge (Y/N); in addition to an inquiry-only condition). Participants also suggested the use of a Time Series Design using multiple cohorts and/or sites using pretest, progress tests, post tests and transfer measures (Clark and Snow, 1975). One advantage of time series designs is that it is possible to modify training to incorporate successful features over time without introducing new threats to validity.

All participants suggested capitalizing on existing training programs such as the Army Graduate Training programs.

References

Burns Jr., W. R., & Freeman, W. D., (2008). Developing an adaptability training strategy and policy for the DoD: An interim report. Institute for Defense Analysis. 4850 Mark Center Drive Alexandria, VA.

Clark, R. E. & Snow, R. E. (Fall, 1975) Alternative designs for instructional technology research, AV Communication Review. 23(4).

Kirschner, P., Sweller, J. and Clark, R. E. (2006). Why minimally guided learning does not work: An analysis of the failure of discovery learning, problem-based learning, experiential learning and inquiry-based learning. Educational Psychologist. 41(2), 75-86.

Singley, M. K., & Anderson, J. R. (1989). Transfer in the ACT* theory. In The transfer of cognitive skills.

Tobias, S., & Duffy, T.D. (2009) (Eds.) Constructivist instruction. Success or failure? New Yorks: Routledge, Taylor, and Francis.

Back to top

Report on the “Games for Change” Meeting

Sigmund Tobias
University at Albany
State University of NY

The sixth annual meeting of Games for Change was held in New York City May 27-29th. A total of about 400 individuals attended. The average attendance at prior meetings of this group was about 350.

The first day consisted of an invitational meeting dealing with evaluation issues, and an all day open meeting called "A Workshop for Making Social Issues Games," attended by about 175 new members ("newbies") of the games community. The newbies were mainly interested in game development and the major theme of the Workshop was that games were a perfectly designed learning environment. Well-known game designers made presentations and they, plus other experienced designers, circulated as resource people to facilitate progress during breakout sessions. Issues discussed at the Workshop dealt with publishing and development paradigms for games, funding sources for the game development, and what developers could learn from of the games they had created. A design breakout session at the Workshop featured brain storming of the steps in game development

The general meeting began on the second day with a keynote address by Nicholas Kristof, a prize winning New York Times journalist who described some truly horrifying events in Darfur. He hoped that social impact games, like Darfur is Dying, could change people's values in a pro-social direction. Kristof suggested that games involving people in causes larger than themselves could promote grass roots movements to put an end to evils like Darfur.

Participants then had a choice of two tracks: strategy and action. The action track included a panel on literacy with Colleen Macklin, John Sharp, Barry Joseph, Mary Flanagan and James Bachhuber. Three programs were presented in the session dealing with games as tools for addressing societal concerns. A succeeding panel in the action track dealt with documentary games with panelists Steve Anderson, Tracy Fullerton, Emily Verellen, Judith Helfand, and Susana Ruiz. This panel discussed some of the problems and potential of documentary games, a new genre in the field.

The first strategy panel consisted of Joseph Kahne and Ian Rowe. Kahne, Mills College, presented the results of a national survey of 1102 students and their parents; only 39 students had not played any games at all. The survey found that, in contrast to widely held negative expectations about games, 76% of the respondents helped others while gaming, 52% reported playing games in which they thought about moral and ethical issues, 44% learned about a societal problem, 43% helped make decisions about how communities should be run, and 40% learned about social issues. The survey also indicated that gaming experiences were closely related to other measures of civic engagement, and that the overall frequency of game playing was unrelated to civic or social isolation. The full report of the survey may be retrieved from www.civicsurvey.org.

In the second strategy panel, Ian Row, Gates Foundation, indicated that the basic Foundation motto was that all lives have equal value. The Foundation wants to explore ways in which games can address major social problems. Specifically, they are interested in ways that games can improve learning and reduce the very high dropout rates among minority students from high school and college.

The second strategy panel dealt with games and assessment, and much of the discussion involved some topics raised at the invitational assessment meeting held during the preceding day. James Gee, ASU, Katie Salen, NYU, Constance Steinkuehler and Kurt Squire, both of the University of Wisconsin were the panelists. Gee started off by urging game designers to teach problem solving in civics contexts rather than facts. Such problem solving assists students in making good choices. Dan Schwartz’s research at Stanford, Gee reported, has shown that people’s choices successfully predicted their knowledge; however knowledge did not predict the choices they made.

Katie Salen reported that rather than model a game on schools, a useful approach would be to build a school modeled after games, an approach currently being investigated at NYU. An active interchange among the panelists, and between the panel and the audience developed enlarging on a variety of issues including the following:

• Assessments should not be limited to "one shot," but should examine a trajectory of learning over time.
  
  
• Assessments should be directed towards learning in communities, which can predict individual learning, rather than at learning alone which does not predict learning in communities.
  
  
• It is desirable to have players assume the roles of both mentor and the student being mentored in games.
  
  
• Giving players the chance to play and edit games helps students see themselves as game designers.
  
  
• Many games teach values and skills that are consonant with what scientists do.
  
  
• Games produce massive amounts of data, therefore, developers must be clear about the critical data in advance, otherwise they will be overwhelmed with data from students' game playing.
  
  
• There is no one assessment that is equally appropriate for all individuals.
  
  

• It is useful to share data about game playing with students who are often eager to improve their playing strategies.

Thursday afternoon was devoted to a "Grow a Game" workshop providing hands on experiences in creating games about social issues. A second workshop dealt with the process of how a game is proposed, designed, play-tested, and evaluated. The final product consisted of two games that were pitted against one another and the best one chosen by players and judges. Another panel discussed the issues of how companies aiming to sell millions of games could be interested in producing games devoted to social causes.

A "Game Expo" completed the second day in which attendees could play some of the newest games developed by members of the community, including some on HIV/AIDS, being in debt, and others dealing with various social issues. An award was presented to the best games in these areas.

The last day of the meeting began with a "Fireside Chat" with Henry Jenkins, MIT, and Jim Gee. The chat was followed by two tracks. One consisted of representatives from foundations discussing how emerging games in the public interest could be funded. Also in this track was another panel with Public/Private partnerships exploring how games that were fun to play could be developed in such a partnership. The second track consisted of a panel discussing how to express ethics through games. A further panel, "A New Designer Mindset," was in tune with the participatory character of 21st century needs. An open conversation with attendees was then scheduled, followed by a discussion dealing with games and the news. A Keynote address by Lucy Bradshaw, Spore Electronic Arts, closed the meeting.

Back to top