to use the teamwork test -- or not? a psychometric evaluation

To Use the Teamwork Test -- Or Not? A Psychometric Evaluation

Janet L. KottkeCalifornia State University, San Bernardino

Kimberly A. FrenchUniversity of South Florida

Rhiannon J. KirchnerCalifornia State University, San Bernardino

Presented to PTC of Southern California, July 23, 2013

2

TKSA Development

• Stevens and Campion (1994; 1999) develop the Teamwork KSA Test – 35 items, multiple choice format (4 options,

dichotomous scoring)• Reviewed teamwork literature – Identified 14 KSAs identified with effective

teamwork

Kottke, French, & Kirchner PTC July 23, 2013

3

14 KSAs → 5 facets

• These 14 KSAs represent– Five facets• Conflict resolution• Collaborative problem solving• Communication• Goal setting and performance management• Planning and task coordination


4

5 facets → 2 dimensions

• Two higher-order dimensions: – Interpersonal KSAs • Conflict resolution• Collaborative problem solving• Communication

– Self-management KSAs • Goal setting and performance management• Planning and task coordination

Kottke, French, & KirchnerPTC July 23, 2013

5

Teamwork KSA Structure


O’Neill, Goffin, & Gellatly, 2012, p. 37

6

Knowledge, Skill, and Ability (KSA) Requirements for Teamwork

I. INTERPERSONAL KSAs A. Conflict Resolution KSAs

1. The KSA to recognize and encourage desirable, but discourage undesirable, team conflict. 2. The KSA to recognize the type and source of conflict confronting the team and to implement an

appropriate conflict resolution strategy. 3. The KSA to employ an integrative (win-win) negotiation strategy rather than the traditional

distributive (win-lose) strategy. B. Collaborative Problem Solving KSAs

4. The KSA to identify situations requiring participative group problem solving and to utilize the proper degree and type of participation.

5. The KSA to recognize the obstacles to collaborative group problem solving and implement appropriate corrective actions.

C. Communication KSAs 6. The KSA to understand communication networks, and to utilize decentralized networks to

enhance communication where possible 7. The KSA to communicate openly and supportively, that is, to send messages which are: (1)

behavior- or event-oriented; (2) congruent; (3) validating; (4) conjunctive; and (5) owned. 8. The KSA to listen nonevaluatively and to appropriately use active listening techniques. 9. The KSA to maximize consonance between nonverbal and verbal messages, and to recognize

and interpret the nonverbal messages of others. 10. The KSA to engage in ritual greetings and small talk, and a recognition of their importance.

II. SELF-MANAGEMENT KSAs D. Goal Setting and Performance Management KSAs

11. The KSA to help establish specific, challenging, and accepted team goals. 12. The KSA to monitor, evaluate, and provide feedback on both overall team performance and

individual team member performance. E. Planning and Task Coordination KSAs

13. The KSA to coordinate and synchronize activities, information, and task interdependencies between team members.

14. The KSA to help establish task and role expectations of individual team members, and to ensure proper balancing of workload in the team.

(Adapted from Steven & Campion, 1994, Table One, p. 505)

7

Sample Question

• When you set work goals for yourself or your work team, what are the best goals to set?

A. Set goals to "do your best." B. Set general and broad goals. C. Set specific and detailed goals. D. Set easy and simple goals.


8

Criterion validity

• Several studies have found predictability of supervisor and observer ratings of team effectiveness (rs range .20 to .56)– Ellis, Bell, Ployhart, Hollenbeck, & Ilgen, 2005– Leach, Wall, Rogelberg, & Jackson, 2005– McClough & Rogelberg, 2003– Morgeson, Reider, & Campion, 2005

• Mixed results in predicting team member peer ratings


9

Questions about predictive validity

• Miller, 2001– Issues raised about Stevens & Campion validation studies

• Team performance was predicted less well (r = .44) than task work (r = .56)

• TKSA predicted team performance only slightly better than did an aptitude test– But: TKSA added incremental variance beyond GMA

• Aptitude data suggest individual aptitude, not teamwork aptitude measured

– Miller’s own study finds no significant relationship between team performance and TKSA scores


10

Questions about psychometrics

• O’Neill, Goffin, & Gellatly (2012) examine the TKSA – Comprehensive analysis• Reviewed all known studies using TKSA• Classic test theory analysis• Confirmatory factor analysis• Exploratory factor analysis• Convergent, discriminant validation• Criterion validity analysis

– Disappointing results all aroundKottke, French, & Kirchner

PTC July 23, 2013

11

Our plan

• Replicate O’Neill CTT & CFA analyses with fresh samples

• Data sources– Had used the TKSA Test for a group project in several

classes (sample 1)– Collected new data to address unexpected finding

from sample 1 that aptitude was negatively related to teamwork interest (sample 2)

– Collected data for a thesis project on teamwork and task interdependence (sample 3)


12

The 3 samples

• Students from a mid-size southwestern University. – Sample 1 = 251 college students

• Cohort groups, upper level undergraduate-level psychology course, term-length group project.

– Sample 2 = 279 college students • Recruited from psychology and business courses

– Sample 3 = 404 college students • From upper-level undergraduate courses in both psychology

and business departments, either a long-term (i.e., several weeks) or term-length group project


13

Work experience of samples

• Although students, vast majority were employed – 96% in sample 1– 82.4% in sample 2– 88.4% in sample 3

• Work experience ranged from means of 5.5 to 7.3 years – SDs ranged from 5.6 to 6.3 years


14

Table 1. Demographic Descriptive Statistics

Sample 1 Sample 2 Sample 3Demographic Characteristic M SD M SD M SDAge 25.1 6.6 23.1 7.0 23.9 6.1Work experience (years) 7.3 5.9 5.5 6.3 5.9 5.6Number of term-length group projects - - 1.8 2.0 - -Number of work group projects - - 4.4 11.9 - -

Gender (% Female) 82.1% 74.9% 65.1%Ethnicity African American 9.2% 11.5% 8.2% Asian American 4.4% 4.7 % 4.5% Hispanic 45.4% 47.3% 45.0% White 35.1% 26.9% 29.7% Other 6.0% 8.2% 11.9%Education Freshman 0.4% 16.5% 0.0% Sophomore 2.8% 14.3% 4.7% Junior 43.4% 33.3% 50.2% Senior 51.0% 34.1% 42.6% Graduate 2.0% 0.4% 0.2% Other 0.4% 0.4% 0.5%%Respondents with work experience 96.0% 82.4% 88.4%Respondents with supervisor experience 40.2% 24.7% -


15

Analyses conducted

• Analytic Approach– In each sample we evaluated

• Classical test theory– Item, subscale, and total measure properties,

» Means, standard deviations, and item-total correlations – Subscale and total measure means, standard deviations, and internal

reliability • Structure using categorical confirmatory factor analysis

(Mplus) – One factor model, two factor model, and a five factor model

» For multi-factor models, both a covaried model (all factors were allowed to correlate), as well as a higher-order model (includes higher-order general teamwork KSA factor)


16

Supplemental

• Supplemental analysis– To ensure psychometric issues are not a result of student

samples, ran classical test analyses on the subset of working respondents from each sample.

– Results from working samples were not substantially different than full samples• For example, average absolute difference in factor and total

means was .11 for sample 1 [SD = .07], .21 for sample 2 [SD = .19], and .14 for sample 3 [SD = .21].

• The average difference in inter-item correlations was .02 [SD = .01])

• Thus, only the results from the full samples are presented here.


17

Table 2.Teamwork KSA Test item-level means, standard deviations, and item-total correlations for all samples.

Sample 1 Sample 2 Sample 3 Subscale Item Statistics M(p) SD ri-f ri-h ri-t M(p) SD ri-f ri-h ri-t M(p) SD ri-f ri-h ri-t

Conflict resolution mean 0.69 0.43 0.14 0.22 0.26 0.64 0.45 0.08 0.22 0.17 0.67 0.44 0.05 0.16 0.17Conflict resolution SD 0.16 0.07 0.09 0.14 0.15 0.19 0.06 0.05 0.11 0.14 0.20 0.07 0.05 0.11 0.12

Communication mean 0.54 0.48 0.13 0.17 0.17 0.54 0.48 0.08 0.13 0.15 0.53 0.48 0.11 0.14 0.16Communication SD 0.13 0.02 0.06 0.06 0.07 0.13 0.02 0.09 0.11 0.11 0.15 0.02 0.07 0.08 0.07Collaborative problem solving mean 0.42 0.47 0.13 0.18 0.20 0.42 0.48 0.14 0.16 0.18 0.40 0.47 0.11 0.14 0.16Collaborative problem solving SD 0.14 0.02 0.09 0.16 0.17 0.13 0.02 0.09 0.10 0.10 0.13 0.02 0.09 0.08 0.09Goal setting & performance management mean 0.60 0.47 0.20 0.22 0.23 0.58 0.48 0.07 0.13 0.19 0.59 0.48 0.19 0.25 0.28Goal setting & performance management SD 0.12 0.04 0.07 0.04 0.05 0.12 0.01 0.07 0.07 0.05 0.11 0.01 0.06 0.02 0.02Planning and task coordination mean 0.64 0.47 0.15 0.18 0.24 0.63 0.47 0.13 0.17 0.23 0.64 0.47 0.15 0.22 0.24Planning and task coordination SD 0.10 0.03 0.02 0.02 0.07 0.12 0.03 0.05 0.09 0.09 0.13 0.04 0.03 0.02 0.05Grand mean 0.55 0.47 0.14 0.19 0.21 0.54 0.48 0.10 0.15 0.17 0.54 0.47 0.12 0.17 0.19Grand SD 0.16 0.04 0.08 0.10 0.12 0.15 0.03 0.08 0.10 0.10 0.16 0.03 0.07 0.08 0.08Notes. M(p) = item means, SD = item standard deviations, ri-f = corrected item-factor correlation,, ri-h = corrected item-higher order factor correlation, ri-t = corrected item-total correlation.

RESULTS: Item means, SDs, item total correlations


18

Table 3. Teamwork KSA Test facet and total means, standard deviations, reliability, and item correlation summary for all samples

Sample 1 Sample 2 Sample 3

No. of Items M SD K-R 20

Inter-item correlation

M SD K-R 20


M SD K-R 20


Teamwork KSA Test Score

M Range M Range M RangeInterpersonal KSAs 25 13.05 3.55 .58 .06 -.12, .32 12.84 3.36 .52 .05 -.17, .23 12.72 3.29 .51 .04 -.15, .19 Conflict Resolution 4 2.75 .97 .25 .09 -.06, .26 2.55 .95 .15 .05 -.01, .14 2.67 .91 .10 .03 -.11, .16 Communication 12 6.49 2.04 .36 .05 -.11, .26 6.53 1.89 .24 .03 -.10, .22 6.41 1.99 .33 .04 -.11, .19 Collaborative Problem Solving

9 3.82 1.71 .34 .05 -.12, .23 3.76 1.72 .35 .06 -.08, .22 3.63 1.64 .28 .04 -.07, .18Self-Management KSAs 10 6.23 1.99 .49 .09 .00, .27 6.06 1.87 .39 .06 -.08, .22 6.11 2.10 .54 .11 -.01, .26 Goal Setting & Performance Management

5 3.01 1.29 .40 .12 .04, .24 2.90 1.16 .17 .03 -.06, .14 2.93 1.29 .38 .11 .03, .23 Planning & Task Coordination

5 3.22 1.20 .30 .08 .00, .18 3.17 1.19 .27 .07 -.05, .22 3.19 1.21 .32 .09 -.01, .26Teamwork KSA Overall 35 19.29 4.84 .69 .06 -.14, .32 18.81 4.69 .65 .05 -.17, .25 18.82 4.70 .67 .05 -.15, .26

More Results: Scale means, SDs, inter item correlation means, KR 20s


19

Confirmatory Factor Analysis Results on following slide


20

Table 4. Summary of confirmatory factor analysis results

Sample Model 2 df CFI RMSEA

Absolute Loading Mean Loading Range

Sample 1 Five factor, covariedb 198.12* 158 .85 .032 .34 -.10 to .83 Revised five factor, covarieda 309.23** 152 .41 .064 .41 -.05 to .92 Five factor, higher order Did not converge Revised five factor, higher ordera 359.39** 154 .24 .072 .44 -.03 to 1.00 Two factor, covaried 202.701* 159 .84 .033 .34 -.51 to .82 Two factor, higher order Did not converge Revised two factor, higher ordera 202.95* 156 .36 .066 .36 -.09 to 1.00 Single factor 202.701* 159 .84 .033 .34 -.10 to .82Sample 2 Five factor, covariedb 211.12 183 .88 .022 .29 -.08 to .58 Revised five factor, covarieda 308.61** 182 .47 .048 .38 -.09 to .71 Five factor, higher order Did not converge Revised five factor, higher ordera 368.40** 182 .22 .058 .38 -.09 to 1.00 Two factor, covaried 212.79 184 .88 .023 .30 -.49 to.62 Two factor, higher order Did not converge Revised two factor, higher ordera 363.73** 182 .24 .057 .32 -.08 to 1.00 Single factor 212.54 184 .88 .022 .30 -.08 to .62Sample 3 Five factor, covariedb 327.42** 213 .78 .036 .33 -.01 to .61 Revised five factor, covarieda 471.13** 211 .51 .055 .40 .01 to .71 Five factor, higher order Did not converge Revised five factor, higher ordera 565.28** 209 .32 .065 .40 .01 to 1.00 Two factor, covaried 345.70** 214 .75 .039 .32 -.55 to .52 Two factor, higher order Did not converge Revised two factor, higher ordera 527.38** 211 .40 .061 .32 .03 to 1.00 Single factor 353.40** 214 .74 .040 .30 -.01 to .52

*p < .01, **p < .001Kottke, French, & Kirchner

PTC July 23, 2013

21

Discussion • To be fair to the test authors

– They recommend the subscales be used for training and developmental purposes only

– In all of their publications, they use only total scores– So, whether the test authors intended a hierarchical structure is not

a certainty• Yet, with their careful development that categorized the 14 KSAs

into specific facets, one would expect meaningful structure• Criterion validity has been found, but issues here as well

– TKSA test correlates highly with general mental ability– Some incremental predictability


22

Conclusion

• Very limited evidence for adequate psychometric properties

• Use with caution– Recognize the test’s limitations


23

Thank you


24

Questions?


25

ReferencesEllis, A. J., Bell, B. S., Ployhart, R. E., Hollenbeck, J. R., & Ilgen, D. R. (2005). An evaluation

of generic teamwork skills training with action teams: Effects on cognitive and skill-based outcomes. Personnel Psychology, 58, 641-672.

Leach, D. J., Wall, T. D., Rogelberg, S. G., & Jackson, P. R. (2005). Team autonomy, performance, and member job strain: Uncovering the teamwork KSA link. Applied Psychology: An International Review, 54, 1-24.

McClough, A. C., & Rogelberg, S. G. (2003). Selection in teams: An exploration of the Teamwork Knowledge, Skills, and Ability test. International Journal of Selection and Assessment, 11, 56-66.

Miller, D. L. (2001). Reexamining teamwork KSAs and team performance. Small Group Research, 32(6), 745.

Morgeson, F. P., Reider, M. H. & Campion, M. A. (2005). Selecting individuals in team settings: The importance of social skills, personality characteristics, and teamwork knowledge. Personnel Psychology, 58, 583-611.

O'Neill, T. A., Goffin, R. D., & Gellatly, I. R. (2012). The knowledge, skill, and ability requirements for teamwork: Revisiting the teamwork‐KSA test's validity. International Journal of Selection And Assessment, 20, 36-52.

Stevens M. J. & Campion M. A. (1999). Staffing work teams: Development and validation of a selection test for teamwork settings. Journal of Management, 25, 207-228.

Stevens, M. J. & Campion, M. A. (1994). The knowledge, skill, and ability requirements for teamwork: Implications for human resource management. Journal of Management, 20, 503-530.


to use the teamwork test -- or not? a psychometric evaluation

Documents

teamwork test

set goals

tksa test

teamwork ksa test

work goals

tksa scores kottke

work team

best goals