development and validation of training themes for jimm operations

Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2009

Development and Validation of Training Themes for Joint, Interagency,Intergovernmental and Multinational (JIIM) Operations

Karol G. Ross

Cognitive Performance Group

Orlando, FL

[email protected]

Anna Grome

Klein Associates Division, ARA

Fairborn, OH

[email protected]

Jim Ong

Stottler Henke, Inc

San Mateo, CA

[email protected]

Meagan Arrastia

Florida State University

Tallahassee, FL

[email protected]

Brooke B. Schaab

US Army Research Institute, JFCOM

Suffolk, VA

[email protected]

David Spangler

Global Innovation & Design, LLC

Chesapeake, VA

[email protected]

ABSTRACT

The contemporary operating environment for U.S. military forces requires a newemphasis on collaboration acrossservices, across government agencies and with non-governmental agencies andhostnations to create global stability.This paper documents an effort to capture the expertise of experienced military and non-military players in theseoperations. Interview findings, operational lessons learned, and previous research form the basis for our investigationinto how best to prepare our forces to plan and execute operations in Joint, Interagency, Intergovernmental andMultinational (JIIM) environments. First, wereviewed documents thatreflected lessons learned or otherinsights intothese types of collaborations. Second, we conducted interviews with military and civilian experts inoperations thatrequired collaboration among the military and other organizations, aswell as host nation officials and citizens. Third,we analyzed those interview transcripts and combined them with the findings of the document review in order toderive themes that expressed the high-level cognitive skills evident in JIIM operations. We found the followingthemes in the document and interview data: Understand the situation within its historical, regional, and culturalcontext; Understand the other participants; Shift perspective; Establish and maintain common ground; Buildcapability to affect the situation; Visualize the operation; Support information exchange; and, Maintain flexibility.We describe our six-step thematic analysis method and the resulting themes. Finally, we report on our themevalidation process with SMEs. The resulting themes currently form the basis for high-level learning objectives aspart of an emerging joint staff-training product for the U.S. Army Command and General Staff College, FortLeavenworth, Kansas.

ABOUT THE AUTHORS

Karol G. Ross is the Chief Scientist for the Cognitive Performance Group. She is currently pursuing research in theareas of cross-cultural competence training and assessment as well as the development of cognitive metrics tosupport training assessment for the defeat of Improvised Explosive Devices (IEDs). She earned herdoctoral degreeinexperimental psychology in 1984 from theUniversity ofTennessee, Knoxville.

2009 Paper No. 9194 of12

Interservice/Industry Training, Simulation, and Education Conference (1/lTSEC) 2009

Meagan C. Arrastia is a former Research Associate at the Cognitive Performance Group. Currently, she ispursuing a PhD in Instructional Systems with an emphasis inCognitive Psychology at Florida State University. Hercurrent research focuses on the assessmentof cross-cultural competence. Meagan holds a BA and an MA in AppliedSociology from the University of Central Florida, Orlando, Florida.

Anna Grome is a Senior Cognitive Scientist at Klein Associates Division of Applied Research Associates. Oneofher core focus areas is uncovering and describing the cultural influences on decision-making and multinationalcollaboration. She has led and supported multiple efforts in this vein for the Army and Air Force, using in-depthinterview methods to uncover how mental models of collaboration and decision-making differ across culture,developing training requirements for crowd control in the Middle East, and developing strategic communications forinfluencing Middle Eastern terrorist networks. She iscurrently a core team member on an effort for the Office of theSecretary of Defense to develop Situated Cultural Training (SCT) for military personnel deployed to Afghanistan.Mrs. Grome holds a M.S. in Industrial-Organizational and Human Factors Psychology from Wright State University,Dayton, OH and a B.A. inPsychology and Spanish from Denison University, Granville, OH.

Brooke B. Schaab has been a Research Psychologist withthe U.S. ArmyResearch Institute for 10years. Her currentwork focuses on improving the capability of military personnel to lead, participate, and make decisions asmembersof joint, interagency, intergovernmental and multinational teams and to enhance sense making, collaboration,decision-making, and adaptation innetwork-enabled operations. Partners in these efforts include Spain, Sweden, andseveral African nations.

Jim Ong leads the development of advanced training and decision support systems and authoring tools at StottlerHenke. He specializes increating authoring tools and intelligent tutoring systems that provide automated coaching,performance assessment, and feedback. Previously, he served in applied research, software product development,software consulting, and systems engineering roles at Bolt, Beranek and Newman, PPD, and AT&T BellLaboratories. Jim received MS degrees in electrical engineering and computer science from the University ofCalifornia at Berkeley and from Yale University.

David Spangler has over 26 years experience in leadership, organization and management and was an AssociateProfessor with over 10 years experience at the post-graduate level, and Master Faculty with over 3 1/2 yearscurriculum and lesson development experience. He was a Navy Fighter and Instructor Pilot with 6.5 yearsexperience, over 3000 flight hours, 560 arrested landings and 20 combat sorties. His areas of specialty includetraining development, modeling & simulation design, and modular education for Simulation Scenario & ExerciseDevelopment; Multinational, Joint, IA Education & Training; Joint Planning & Assessment; Design & SystemsPerspective; Curriculum Development; Military Acculturation.

2009 PaperNo. 9194 Page2 of 12


Development and Validation of Training Themes for Joint, Interagency,Intergovernmental and Multinational (JIIM) Operations

Karol G. Ross

Cognitive Performance GroupOrlando, FL

[email protected]

Anna Grome

Klein Associates Division, ARA

Fairborn, OH

[email protected]

Jim Ong

Stottler Henke, Inc

San Mateo, CA

[email protected]

INTRODUCITON

The contemporary operating environment for the U.S.military forces requires a new emphasis oncollaboration across services, across governmentagencies and with non-governmental agencies and hostnations in order to create global stability in the interestof national security. While U.S. national objectivesdrive operations, our interests are increasinglydependent on supporting the security and stability of awide range of regions and nations in which we findourselves conducting operations. The skills todetermine relevant objectives and take effective actionsin such a collaborative environment have long been apartof the U.S. military capabilities, butthese skills arenow more in the foreground of operations and arerequired of a wider rangeof personnel from the tacticalto the strategic level.

New and emerging doctrine directs the nature of theseinteractions for the U.S. military. (See for example U.S.Army Field Manual 3-07 "Stability Operations,"October, 2008; Joint Publication 3-07.3, "PeaceOperations," 17 Oct 07; and DoD Directive 3000.05,"Military Support for Stability, Security, Transition,and Reconstruction (SSTR) Operations," 28 Nov 05).While this doctrine provides guidance and structure, itdoes not capture the expertise required to apply thatdoctrine successfully. The purpose of this effort was tocapture the expertise of experienced members of themilitary and combine it with the expertise of non-

2009 Paper No. 9194 of12

Meagan Arrastia

Florida State University

Tallahassee, FL

[email protected]

Brooke B. Schaab

US Army Research Institute, JFCOM

Suffolk, VA

[email protected]

David Spangler

Global Innovation & Design, LLC

Chesapeake, VA

[email protected]

military players in these settings, as well as withoperational lessons learned and previous research inorder to insure that training and education prepares ourforces to plan and implement Joint, Interagency,Intergovernmental and Multinational (JIIM)environments.

In these types of operations, expertise is concentratedwithin and dispersed across certain elements of themilitary forces such as Civil Affairs, units who havesuccessfully engaged in counterinsurgency operationsin Iraq, members of Provincial Reconstruction Teamsin Afghanistan, and those who have served in Joint orState Department positions. Additionally, expertise hasdeveloped in non-government organizations that arepart of these collaborative efforts. Our approach tounderstanding this expert performance was to conductinterviews across this range of experienced peopleusinga critical incidentapproach and then to performathematic analysis to capture the high-level cognitiveskills in the form of themes that can inform the focus oftraining and education.

Theme-based training of cognitive skills is a successfulstrategy employedby researchersworking in the area ofcognitive psychology for the military. The concept ofusing themes to guide cognitive development throughsituated learning has grown from the constructivistapproach to instruction.


The Cognition and Technology Group at VanderbiltUniversity (CTGV; 1990) presented one of the firstemployments of theme-based instruction usingtechnology in their model of situated instruction. Theycreated an environment for students to experience aproblem from multiple perspectives or themes. Theenvironment was for fifth-grade students to teach themall aspects of story writing (e.g., initiating events,character development, differentiation of protagonistand antagonist, setting) and, thus the word themes—acommon concept in literature, was used to capture thehigh-level concepts to be taught. They explored onespecific movie "Young Sherlock Holmes" on videodisk using the technology to access scenes repeatedlyfrom many viewpoints. The group receiving theme-based education excelled beyond the control group thatstudied rules of writing in a more linear fashion withcurrent, accepted reading and writing programs. Thegroup engaged in the theme-based instruction wrotestories with more elements; produced plots that linkedcharacters and events to goals more often; used targetedvocabulary more frequently; and produced higherqualityclassroomdiscussions.

The goal of such instruction is to go beyond superficialfamiliarity with concepts and facts to mastering theconceptual complexity of an area. Spiro, Coulson,Feltovich, and Anderson (1988) produced a model ofCognitive Flexibility to guide learning which also putforth the concept of a deeper level of cognitive insightgained by theme-based exploration of situations orscenarios. They asserted that this method of learningsupported the ability to spontaneously restructure theknowledge gained in adaptive response to changingsituations. Spiro, et al. (1992b) demonstrated thisapproach with adult learners in the area of tacticalthinking. They used one case study, the battle ofChancellorsville—a case study typically used in U.S.Army tactical education—to support theme-basedlearning. Their efforts focused on demonstrating adeeper conceptual understanding of tactics bettergeneralization to new settings by using a theme-basedstructure to examine a scenario or case.

The purpose of thematic instruction, then, is to aid thelearner in going over the same problem space fromdifferent viewpoints. This instructional strategypromotes cognitive flexibility in a domain of practiceand avoids counter-productive training in introductorytraining and in some higher-level learning. By counterproductive training, we refer to training that inhibitstransfer of knowledge to field performance. Reasonsfor failure include oversimplification of concepts, linearpresentation of material ignoring the inter-related

2009 Paper No. 9194 of12

nature of concepts, using one exemplar leading to astudent perception of one right answer for complexproblems, and using simple analogies for complexsystems (Spiro, Feltovich, Jacobson, & Coulson,1992a). Theme-based instruction when applied as amethod to help the student explore situations fromdifferent perspectives produces a more cognitivelycomplex understanding of a domain more quickly andtransfer of concepts to field performance in a moreflexible manner.

This approach has been successful in military training.During the development of the "Think like aCommander" (TLAC) training environment, a numberof high-level cognitive skills common acrossperformance of expert tacticians were summarized toguide practice in this domain. The approach of TLACis to explore a situation from these multipleperspectives or themes (Ross, & Lussier, 1999). Thissuccessful approach to training thinkingskills (Lussier,Shadrick, & Prevou, 2003) is also the basis for anotherresearch effort to produce training for military crisismanagement thinking skills in the Red Cape tool(Shadrick, Schaefer, & Beaubien, 2007).

While a theme-based approach has been successful inmilitary training in limited applications, the method forthematic analysis is not clear in the literature. Thematicanalysis is a qualitative method for encodinginformation, usually interview data, although othersources may be included such as documents. Theresults are a list of themes, lists of indicators, ordescriptive models. Theme-based training as anapproach to situated or scenario-based training andeducation is dependent on generating acceptablethemes that reflect the high-level cognitive skills in adomain. Many researchers and training developers donot accept or understand the qualitative analysis that isthe basis of such training. This problem is, in part, dueto the lack of presentation of the method of analysis.This deficiency affects the ability of the research anddevelopment community to duplicate successfultraining approaches based on themes.

In the qualitative research literature, "[t]hematicanalysis is widely used, but there is no clear agreementabout what thematic analysis is and how you go aboutdoing it" (Braun, & Clarke, 2006, p. 79). There are anumber thematic analysis methods (such asconversation analysis, interpretative phenomenologicalanalysis, discourse analysis, and narrative analysis).Thematic analysis is generally used to pull meaningfrom the data in order to understand a phenomenon orspecific aspects of a phenomenon that are


psychological (such as the experience of choosingcosmetic surgery or of going through a serious illness)or social in nature (such as patterns of drug abuse in aparticular segment of society). Because theme-basedcognitive training has been successful, if limited, it isimportant to clarify how the method for developingthemes.

THEME DEVELOPMENT: STUDY 1

Method

Our method consisted of three primary activities. First,we identified relevant documents that reflected lessonslearnedor other insights into militaryjoint, interagency,intergovernmental and multinational operations.Second, we conducted a series of interviews withsubject matter experts (SMEs) whom we had identifiedas experienced in operations that required collaborationbetween the military and other organizations, as well ashost nation officials and citizens. Third, we analyzedtranscripts from those interviews to derive themes thatexpressed the high-level cognitive skills evident in theexperiences. To represent our findings, we combinedinterview findings with insights gained from thedocuments we had reviewed to produce a table ofthemes and a model ofhow the themes interact.

Research QuestionA theme captures an important aspect of the data inrelation to the research question and displays thefindings in a patterned manner across the data set(Braun, & Clarke, 2006). Our research question waswhat are the high-level cognitive skills that underliesuccessful performance in JIIM operations?

Participants and DocumentsWe recruited interview participants with experience inJIIM environments. Our goal was to compare multipleperspectives at tactical and operations levels.Participants in the military or associated with militaryorganizations were volunteers. We provided monetarycompensation for SMEs from other organizations. Weinterviewed a total of 16 military or military-relatedSMEs (one of which was not used for analysis); threeAfrican nationals, one affiliated with a non-governmentorganization and two with a government healthorganization; and five SMEs with Department of Stateexperience for a total of23 interviews.

As for documents reviewed, we included a report thatmembers of our project team had produced on lessonslearned in JIIM operations as an earlier part of this

2009 PaperNo. 9194 of12

effort (Agrait, & Loughran, 2007). This documentreviewed lessons learned, previous interviews,workshop documentation, and relevant documents tosynthesize the state of performance in JIIM settings interms of players, processes, and environments. Inaddition, we identified several other documents thatprovidedinsights into performance that is contingentoncollaborating with host nation government andcivilians, other military services, and other agencies.

Specifically we reviewed seven documents:1) The JIIM lessons learned document2) Transcripts of three days of meetings of the HASE(Healthy Africa Scenarios Exercise) Workshop inGhana, Africa held January 21-25, 20083) A briefing on the Joint, Interagency,Intergovernmental and Multinational Planner's Coursefrom the Joint Forces StaffCollege4) A report on cognitive challenges in operationsotherthan war (Miller, et al., 2003)5) A report that reviewed the cognitive challenges ofworking across military organizations (O'Dea, et al.,2006)6) A report on the elements of cross-culturalcompetence required for military operations (Ross,2008)7) A draft report on modeling cross-culturalcompetence in the U.S. Army (McCloskey, Grandjean,& Ross, in publication)

Data Collection Procedure

We used a semi-structured interview protocol was usedthat was based on the Critical Decision Method (CDM)(Crandall, Klein, & Hoffman, 2006; Hoffman,Crandall, & Shadbolt, 1998). We captured digitalrecordings of all interviews, and all interviews weretranscribed for analysis.

Analysis PlanThe guiding principle in our analysis was to find theelements of performance that are key aspects ofsuccessful planning and operations in JIIMenvironments. Key aspects were not determined by asimple prevalence count, i.e., how many times certainperformance elements were mentioned, but by theemphasis placed on how the operations areaccomplished or how they fail, i.e., expert strategies.Existing documents that had previously examinedoperations in terms of expertise or lessons learnedhelped us determine the emphasis to place on elementsof performance.

Our approach was inductive, i.e., the themes identifiedemerged from the data gathered specifically for this

Interservice/Industry Training, Simulation, andEducation Conference (I/1TSEC) 2009

project or to reports that are based on data aboutrelevant performance. In this approach, the themesmaybear little relation to the actual questions asked in theinterviews. We developed questions to help usindirectly assess the elementsof expertise by gatheringrich examples of performance, insights, and strategiesthat drove performance. One assumption of theresearch is that as expertise develops, people are notgenerally able to articulate the how and why of theircognitive performance. Therefore, the analysis usesthespecifics of the data, but the data itself has beencollected to expose the cognitive challenges andstrategies inherent in situations without directly askingparticipants why they didthings theway they did.

The inductive process does not try to fit the data into apre-conceived framework, but creates the frameworkfrom the data. The process also provides a richdescription of thecognitive challenges andstrategies asopposed to a description of the procedures involved inplanning and operations. Analysis requiresinterpretation of specific data to general themes. Ingeneral, we coded specifics into categories within eachindividual data item (interview or document) and thensummarized across the data set.

Thefirst stepin the process wasto readthe data set toimmerse the analysisteam in the findings. The researchquestion informs how reading proceeds. The analysisteam made notes of interesting ideas in the data anddocumented those that we could possibly convert intocoding categories. Review of the selected documentswas interspersed with reviews of the interviewtranscripts. Two analysts experienced in thematicanalysis for cognitive performance and experienced inthe subject area conducted the first step. The outcomewas a set of potential codes, viewed as preliminarythemes, from each analyst and notes to connect eachpotential theme to the interviews and documents thathad suggested the code. During this step, moreresearchers typically generate more codes than will beretained during the final analysis.

The second step was to generate initialcodes. In thiscase, the codes are preliminary themes of expert high-level cognitive skills. The two analysts independentlyreviewed each other's codes and rationale for each andthen discussed the overlap and wording for each code.

2009 PaperNo. 9194 of12

The outcome of this step was an agreed upon set ofcodes that were to serve as preliminary themes.

The third step was to convert the preliminary themesinto a representation to help the team understand thenature of each theme. We constructed a three-columntable to present an initial name for each theme, a one-paragraph definition of the theme, and a list of thecognitivechallenges associatedwith the theme.

The fourth step was to review the themes moreexhaustively against the data set to identify areasneeding refinement. In this step, the analysis team reread the data set. A third analyst joined the team andthe entire team reviewed each data item completely.Each item was coded according to the themes that hadbeen developed,and an additional field was created for"other" themes or interesting elements that emergedduring thecomplete review. The number of instances ofa theme found in the data set, or prevalence, does notnecessarily mean a themeis morecrucial.Our inclusionof other documents in the data set allowed us toexamine the nature of performance in JIIMenvironments in general across a wide range of dataand previous analysisin order to makejudgmentsaboutthe criticality of any one aspect of performance andjudge whether to retain inclusion of a theme in therepresentation.

The fifth step was to refine the names of each themeand the corresponding definitions and cognitivechallengesfor each. Our goal was to have a name foreach theme that was easy to remember and to createdefinitions and explanations of cognitive challengesdrawn from the data, i.e., using the words of theexperienced interview participants or findings inprevious documentation to the extentpossible to defineand explain the themes.

The sixth step was to create a model of expertise inJIIM planning and operations that reflected the high-level cognitive skills of experienced practitioners asopposed to a behavioral analysis, i.e., proceduralrepresentation. Our goal was to reflect the flow ofperformance and inter-relationship among the themesas shown in Figure 1.

Interservice/Industry Training. Simulation, and Education Conference (I/ITSEC) 2009

1. Understanding thesituation within its

historical, regional,and cultural context

- Learn to read these

types of situationsand know who the

typical players are.

8. Maintain flexibility across interactions to understand andadapt new approaches and to be resilient when attempts tounderstand others or implement plans arc not immediatelysuccessful.

Provide

knowledgeto

2. Understanding theother participants inthe specific situation- Who arc they?How do they dobusiness? What has

their role been in this

situation?

Provide

knowledgeto

3. Shift perspective -Understand how others

see the situation and

your organization.

Must be implementedas you •

4. Establish and

maintain common

ground - Whatmutually beneficialgoals can you workon?

6. Visualize the

operation including2d and 3rd order

consequences.

Provide

knowledoejto

5. Build the

capacity to affectthe situation byusing all assetsand expertise

Is the basis for effective

7. Support olinformation

exchange

Figure 1: Model of High-Level Cognitive Skills Required for JIIM Operations

Results

We describe the eight resulting themes below in termsof a progression in complexity of performance frominitial understanding of the situation to maintainingflexibility during operations.

1. Understand the situation within its historical,regional, and cultural context - Operations willtake place in diverse settings requiring theindividual to have a framework for entering andunderstanding widely diverse, new situations,faking this mental model into the setting will bethe factor that allows the individual to becomemore immediately effective. A clear framework forgrasping a new area of operations and the eventsleading up to the current situation provides a stablebasis for assessment and decision-making.Commonly, area briefings prior to deployment donot bring the situation to life or provide a goodunderstandingof the dynamics involved, especiallyif the individual brings no framework for absorbingthe information to the mission introduction.

2009 Paper No. 9194 of 12

Understand the other participants - The secondframework that an individual should take into a

JIIM environment is a basic understanding of whothe various players are that typically work in thatsetting. Knowing the types of organizations andentities that can be involved will facilitate

identification and understanding of the playerscurrently involved. The individual should alsounderstand how these organizations and groupstypically function. There will be differences in howthey do business such as differences in work paceand work hours; integration into the community;maintenance of clear cut roles and responsibilitiesversus diffuse roles; communication styles: metricsfor progress: comfort with and availability oftechnology: authority structures; and whether theyconcentrate on tasks or relationships to do business.For example, one item of critical importance in anysituation is to understand who has the decision

making power and what the process is for makingdecisions in the organizations and other groupswith whom one must negotiate or collaborate. Our

Interservice/Industry Training, Simulation,and Education Conference (I/ITSEC) 2009

forces often seem to disregard or discount theprocesses and goals of other organizations orgroups, which negate our ability to collaborate,negotiate or leverage expertise.

3. Shiftperspective - Once the individual has enteredthe situation, he or she can facilitate collaborationby using the ability to shift perspective in order tosee the situation from another person's point ofview. Shifting perspective allows one tounderstand, predict, and influence behavior andfoster communication. This ability also includes theneed to be aware of how you and your organizationappear to other organizations and the hostnation/region. One must understand others'rationale for decision-making, consider their intent,their priorities, patterns of living, and long-termgoals. A wide range of our forces in the JIIMsetting must more routinely apply this skill, whichis typical of U.S. Army Civil Affairs or SpecialForces.

4. Establish and maintain common ground - As abasis for collaboration and stability, our forcesmust establish mutual goals and interests. This isthe basis for unity of effort in the JIIM situationwhere military unity of command does not apply.Building in the time necessary to find commonground, and actually discovering it becomes asignificant challenge during operations. Justfocusing on one's own organization fits manypeople's comfort zone. Self-regulation to maintaincontrol and openness when dealing with otherstakes practice.The resultingrelationships create thebasis for establishing common goals andcoordinated actions. The ability to shiftperspectives is an ongoing requirement whileworking to establish common ground.

5. Build capability to affect the situation - Once anunderstanding of the situation and the players isbeginning to develop, the staffor teamcanbegin tobuild capabilities across organizations and otherentities that will serve mutual goals. The basis forenhancedcapability is to know and use all assets toaddress common goals much like in any othertactical situation. The difference is that assets inJIIM environment require sustained effort toidentify and use. The diverse set of resources andexpertise that may exist in your own organizationand other participant organizations is often notexplicit. They can go untapped if not deliberatelysought out. Explore who has expertise around

2009 PaperNo. 9194 of12

different issues or problems. Recognize the militarycan bring technical skill and disciplined decisionmaking to the situation. Use those strengths without"shutting out" others who have ownership in thesituation. Recognize the military often does nothave knowledge about what technologies andprocesses work best locally and these must bediscovered.

6. Visualize the operation - Anticipate the need totransition to the next phase including the resourcesneeded for different players. Only by understandingthe other entities in a situation, including theirstrengths and interests, can an individual and teamvisualize an operation and potential consequences.Again, the tactical skill of visualizing an operationis relevant to the JIIM environment but is more

complex and built on collaborators outside one'sorganization know, need and are in the process ofdoing.

7. Support information exchange - Differentorganizations have varying information needs,priorities, and sense of urgency for informationsharing. One must understand that the methods andchannels of communication differ across

organizations such as method of presentation; whoshares information; who has authority forinformation sharing; and how complete informationmust be to support decisions. Communicationstyles impact information exchange and can createa great deal of frustration and use up huge amountsof non-productive time when they are in conflict.Examples of differing communication styles areflexible versus standardized, embedded inrelationships versus "all business" focus, limits onwho shares information and how. Opportunities andlimits to information exchange can vary andinfluence planning and execution. Seek to createaccess to information sharing rather than expectingothers to conform to one's own organizationalexpectations.

8. Maintainfiexibility - The most important aspect ofsuccess in JIIM environments is flexibility. Whenthe U. S. military is in a situation where they arenot in charge, such as in a JIIM setting, thetendency may be to attempt to become morecontrolling. One must recognize when an approachor stance in a situation is not working, and bewilling to adapt it. The themes leading up to"maintain flexibility" are the basis for flexibility inthe JIIM environment.

Interservice/Industry Training, Simulation, andEducation Conference (I/ITSEC) 2009

VALIDATION OF THEMES: STUDY 2

Method

Validation of the themes served to help us understandthe eight themes reflect successful operation in a JIIMenvironment. The methods for this study consisted oftwo main parts, one quantitative and one qualitative.First, we developeda survey instrument consistingof 54items from the previous interviews with JIIM SMEs inorder to validate the eight themesdiscovered.The itemsare statements describing the skills that make up eachtheme with the answer choices displayed in a 5-pointLikert scale ranging from "very important to missionsuccess" to "very unimportant to mission success." Inaddition to the survey, we conducted two focus groupsto solicit feedback about the themes from individualsthat have actually had to employ such skills in JIIMenvironments. Feedback about the themes in the wordsof the SMEs added richness to the data from the surveyinformation. We analyzed the survey data to determinewhether the themes were valid to SMEs. In addition, weanalyzed the focus group transcripts for the denial oraffirmation of the validity of the themes.

Research QuestionThe purpose of the second study was to explore theeight themes that emerged from the previous interviewsand to test the validity of the themes. Our main tworesearch questions are "Are these themes valid in theJIIM environment?" and "Given these themes, what aresome incidents in which these were crucial to missionsuccess?" The intention of the aforementioned surveywas to answer the first question pertaining to thevalidity of the themes and the objective of the focusgroup wasto answer the second question.

ParticipantsOur goal was to test the validity of theme by workingwith a sample of people with many JIIM experiencesand interactions. In order to achieve this goal, werecruited Civil Affairs Officers. We surveyed 18reservist Civil Affairs Officers from the 350th CivilAffairs Command, at the Army Reserve Center inPensacola, FL. The mean age of the sample was 47years and the average number of years in service was23. The sample consisted of three Colonels, eightLieutenant Colonels, two Majors, three Sergeants FirstClass,one Master Sergeant,and one Staff Sergeantwith72% of the sample being Officers. Of those eighteenparticipants, six volunteered to participate in a focus

2009 Paper No. 9194 of12

group based on the extent of their Joint Task Forceexperience.

Data Collection Procedure

All participants signed informed consent forms, whichwe collected before the administration of the surveys, aswell as the beginning of the focus groups. A semi-structured interview protocol was used for the focusgroups whichconsistedof first describingthe themesasstated in the results of Study 1 followed by probingquestions to determine the importance of each themeand the relevance of the statements used to describetheme. We elicited critical incidents in which thethemes were crucial in the success of the mission. Wecreated digital recordings of both focusgroupsand bothwere transcribed for analysis.

Analysis PlanThe aforementioned research questions dictated theanalysis in this study. The first question is whether thethemes found are valid. In order to answer this questionwe analyzedthe surveydata using the following steps:

1. We entered the data into the SPSS data-analysissoftware.

2. We cleaned and checked the data for any errors.This purpose of this step was to check for anyhuman-errors that might have occurred during dataentry.

3. We calculated the mean and standarddeviationforeach question. We constructed a table listing thefrequency of each answer choice, the mean, and thestandard deviation of each question.

4. We combined the individual questions into theirthematic groups. We designed each item to reflectan aspect of one theme. We assembled the itemsback into a group representing the theme theydescribed in order to report on themes and notindividual questions.

5. The mean answer choice and standard deviationfor each theme was calculated. This indicated theconsensus of the importance of each theme tomission success within a JIIM environment. We

constructed a table displaying the percentage ofparticipants that found the theme important basedon the average response across relevant items foreach theme.

Interservice. Industry Training. Simulation, and Education Conference (I/ITSEC) 2009

For the qualitative portion of the analysis, the secondresearch question guided the procedure. The researchquestion was to identify examples of the differentthemes from the incidents reported by the SMEs. Inorder to identify these incidents we followed threesteps:

/. We read the themes to re-familiarize eachresearcher with the definitions.

2. We read the transcripts of the focus groups tofamiliarize the researchers with the data.

3. We analyzedeach transcriptfor each theme, one ata lime. We coded each incidents and catalogued itunder each theme relevant to the incident

described.

Results

Survey ResultsThe participant answered each item with a number fromone to six with one being "very important" and sixbeing "very unimportant." A theme was "important" ifthe with mean of its items ranged from 1.1 to 2.1. Therate of importance for every theme ranged from 91-100% of the respondents giving it an answer of 2.1 orlower. Shifting Perspective was the lowest rankedtheme, being important to 91% of the sample.MaintainingFlexibility was the most important theme—ranked as important by 100% of the sample. Thepercentage of participants that found each themeimportant is shown in fable 1 below. Although thesample size does not allow for generalization of theresults to the U.S. Army population, the tindings doindicate a trend in which these themes are applicable forJIIM performance from the pointof viewof SMEs whowere very experienced in these settings at both tacticaland operational levels.

Focus Group ResultsWe conducted thematic analysis on the Focus Groupdata to derive examples of each theme for later use intraining. We did not code the themes for frequency ineach focus group. Instead, we identified the situationsbased on emphasis given to a particular theme. Thismeans that even though an example given addressedmore than one theme, we identified the mostpronounced theme related to the incident. We identifiedincidents in the focus groups that validated each of theeight themes as important in critical situations.

2009 Paper No. 9194of 12

Table 1: Survey Validation of Themes

Theme Related Total % of

Item Items ParticipantsNumbers Validating

Themes

Understand the 1-9 9 95.1%

situation within

its historical.

regional, andcultural context

Understand the 10-16 7 96.8%

other

participantsShift 17-24 8 91.0%

perspective

Establish and 25-31 7 99.2%

maintain

common

"round

Build capacity 32-38 7 95.2%

to affect the

situation

Visualize the 39-44 6 94.4%

operation

Support 45-49 5 91.1%

information

exchange

Maintain 50-54 5 100%

Flexibility

DISCUSSION

Despite some successful applications of theme-basedtraining in the Army to teach intermediate-levelcognitive skills, there is little documentation of how toderive themes in new competency areas. Themes ofexpertthinking, suchas the ones we havederived in thisproject, support the development of scenario-basedlearning designs. Such designs can accelerate theacquisition of knowledge and boost the learner'scapacity to engage meaningfully in practice situationsearlier than might be expected. Theme-based learningoffers an opportunity to develop mental frameworksthrough practice on how to approach a situation as anexpert. We. as a community, are not leveraging thisavenue to accelerate expertise and adaptiveperformance in part because it is difficult to discoverand organize themes in an area of competency due tothe lack of guidance and examples. Our study providesan exampleof a process for developing themes.

Interservice/Industry Training, Simulation, and EducationConference(I/ITSEC) 2009

We are currently using the themes we identified in thisproject to develop training for the U.S. ArmyCommandand General Staff College, Fort Leavenworth, Kansas.This training embeds practice in areas of each themewithin a range of stability operation vignettes. Thethemes will support acceleration of performance byproviding a framework for how to employ StabilityOperations doctrine. The learner encounters informationabout Stability Operations—mission organization,procedures, and processes within the context ofsuccessful, expert strategies.

The themes provide a framework in the training toreinforce habits of thought that are consistent withexpertperformance to guidedeliberate practice of thesethought processes. In this way, the training acceleratesprogression toward expertise. (See, for example,Ericsson, 2008 and Lussier, 2008). For the CGSCcourse, a heterogeneous group of students will use theJIIM training. Our goal is to provide a collaborativeframework for the intermediate and advanced student toallow them to practice the critical cognitive skills thatcreate success. At the same time, we want to acceleratethe acquisition ofentry-level knowledgeso that studentsnew to these complex environments can quicklyconstruct a framework of what is important for success.Too often training for entry-level knowledge is boringand decontextualized. These methods result in a lack ofretention and a lack in the student's ability to generalizeknowledge to new settings and to higher-level learning,essentially requiring the student to re-learn introductoryinformation later in the education or training process.Our goal is to allow different levels of students toparticipate in and benefit from demanding classexercises following our computer-based tutorials basedon a common framework for expert performance.

ACKNOWLEDGEMENTS

The authors would like to thank Partners InternationalFoundationwho organized the Health African ScenariosExercise workshop and shared workshop data with us.We also want to thank the faculty at Joint Forces StaffCollege, members of the 91st Civil Affairs Battalion atFt. Bragg, the 350lh Civil Affairs Command, Pensacola,Florida, and others who shared their expertise andperspectives on Joint, Interagency, Intergovernmental,and Multinational stability and support operations. TheArmy Research Institute funded this research under acontract with Stottler Henke Associates, Inc., and it wasconducted under the direction of Dr. Brooke Schaab.

2009 PaperNo. 9194 of12

REFERENCES

Agrait, R. & Loughran, J. J. (2007). Lessons Learnedfrom Joint, Interagency, International andMultinational (JIIM) Workshops and Papers. ViennaVA: ThoughtLink, Inc.

Braun, V., & Clarke, V. (2006). Using thematic analysisin psychology. Qualitative Research in Psychology,5,77-101.

Cognition and Technology Group at VanderbiltUniversity (CTGV) (1990). Anchored instruction andits relationship to situated cognition. EducationalResearcher, 19(6), 2-10.

Crandall, B., Klein, G., & Hoffman, R. R. (2006).Working minds: A practitioner's guide to cognitivetaskanalysis. Cambridge, MA: The MIT Press.

Ericsson, K. A. (2008). Accelerating Learning andAcquisition of Expert Professional Performancethrough Deliberate Practice. Presentation at the DoDAccelerated Learning Technology Workshop. Held22-23 July 2008, Alexandria, VA.

Hoffman, R. R., Crandall, B. W., & Shadbolt, N. R.(1998). Use of the critical decision method to elicitexpert knowledge: A case study in cognitive taskanalysis methodology. Human Factors, 40(2), 254-276.

Lussier, J. W. (2008). Adaptive Thinking TrainingforArmy Captains. Presentation at the DoD AcceleratedLearning Technology Workshop. Held 22-23 July2008, Alexandria, VA.

Lussier, J. W., Shadrick, S. B., & Prevou, M. I. (2003).Think Like a Commander prototype: Instructor'sguide to adaptive thinking. (Research Product 2003-02). Alexandria, VA: U. S. Army Research Institutefor the Behavioral and Social Sciences.

McCloskey, M., Grandjean, A., & Ross, K. (inpublication). Assessing learning and development inArmy cross-cultural competence. (Phase 1 SBIRReport.) Alexandria, VA: U. S. Army ResearchInstitute for the Behavioral and Social Sciences.

Miller, T. E., Phillips, J. K., Battaglia, D. A., Wiggins, S.L., Baxter, H., Mills, J. A., & Ross, K. G. (2003).Collaborative online training for operations otherthan war (OOTW) (Final Technical Report forContract #F33165-00-C-6003 Air Force ResearchLaboratory, Mesa, AZ). Fairbom, OH: KleinAssociates Inc.

O'Dea, A., Ross, K.G., McHugh, A, Phillips, J.K.,Throne, M.H., & McCloskey, M., & Mills, J.A(2006). Global teams: Enhancing theperformance ofmultinational staffs through collaborative online

Interservice/Industry Training, Simulation, and EducationConference(I/TTSEC) 2009

training (Research Report 1849, AD A469425).Alexandria, VA: U.S. Army Research Institute forthe Behavioral and Social Sciences

Ross, K. G. (2008, May). Toward an operationaldefinition of cross-cultural competence frominterview data. Patrick AFB, FL: Defense EqualOpportunity Management Institute (DEOMI).http://www.deomi.org/CulturalReadiness/CulturalResearch.cfm#R

Ross, K. G., & Lussier J. W. (1999). A Training Solutionfor Adaptive BattlefieldPerformance. Proceedings ofthe Interservice/Industry Training, Simulation &Education Conference (I/ITSEC), 1999. Orlando FL.

Shadrick, S. B., Schaefer, P. S., & Beaubien, J. (2007).Development and content validation of crisisresponse training package Red Cape: Crisis ActionPlanning and Execution (Research Report 1875).Alexandria, VA: U. S. Army Research Institute forthe Behavioral and Social Sciences.

2009Paper No. 9194of12

Spiro, R. J., Feltovich, P. J., Jacobson, M. J., & Coulson,R. L. (1992a). Cognitive Flexibility, constructivism,and hypertext: Random access instruction foradvanced knowledge acquisition in ill-structureddomains. In T. Duffy, & D. Jonassen (Eds.).Constructivism and the Technology of Instruction: AConversation (pp.57-75). Hillsdale, NJ: LawrenceErlbaum Associates.

Spiro, R. J., Feltovich, P. J., Coulson, R. L., Jacobson,M. J., Durgunoglu, A., Ravine, S., & Jehng, J.(1992b). Knowledge acquisition for application:Cognitive flexibility and transfer of training in ill-structured domains (ARI Research Note 92-21).Alexandria, VA: U.S. Army Research Institute for theBehavioral and Social Sciences

Spiro, R. J., Coulson, R. L., Feltovich, P. J., & Anderson,D. K. (1988). Cognitive flexibility theory: Advancedknowledge acquisition in ill-structured domains. InThe Tenth Annual Conference of the CognitiveScience Society. Hillsdale, NJ: Lawrence ErlbaumAssociates.