test reviews - mentismentisglobal.com/wp-content/uploads/hogan_secure... · the test authors...

TEST REVIEWS

Hogan Development Survey [Revised]

www.mentisglobal.com Distributed by Mentis [email protected]

Hogan Development Survey [Revised]Purpose

Designed to “assess eleven common dysfunctional dispositions.”Population

Ages 16 and over.Publication Dates

1995-2009.Acronym

HDS.Scores, 11

Excitable, Skeptical, Cautious, Reserved, Leisurely, Bold, Mischievous, Colorful, Imaginative,Diligent, Dutiful.

AdministrationGroup.

Price DataAvailable from publisher.

Foreign Language EditionsAvailable in 42 languages: Czech, French Canadian, French Parisian, German, Slovak, Spanish,Castilian Spanish, Brazilian Portuguese, Danish, Turkish, UK English, US English, Italian, Swedish,Dutch, Norwegian, Indian, Greek, South African, Kenya, Bahasa Malaysian, Bahasa Indonesian,Japanese, Korean, Icelandic, New Zealand English, Simplified Chinese, Traditional Chinese, Polish,Russian, Finnish, Romanian, Bulgarian, Thai, Australian, Greek English, Arabic, Estonian, Hungarian,Macedonian, Serbian, Vietnamese.

Time(15-20) minutes.

CommentsAvailable for online administration.

AuthorsRobert Hogan and Joyce Hogan.

PublisherHogan Assessment Systems, Inc.

Cross ReferencesFor reviews by Glen Fox and E. Scott Huebner of an earlier edition, see 14:168.

2All Mental Measurements Yearbook test reviews are copyrighted by the Buros Center for Testing. Reviews may be printed for individual use only,and may not be otherwise duplicated or distributed without written permission. Please refer to the Buros website for further information aboutcopyright and permissions: http://www.buros.org/copyright-and-permissions.


http://www.buros.org/copyright-and-permissions

REVIEW 1 OF 2

Review of the Hogan Development Survey [Revised] by STEPHEN AXFORD,Executive Officer for Special Services, Falcon School District 49, ColoradoSprings, CO, and Adjunct Professor, University of Colorado at Colorado

Springs, Colorado Springs, CO:

DESCRIPTION

The Hogan Development Survey (HDS), based on Socioanalytic Theory, is a multifactor, group-administered measure of “interpersonal competencies” (manual, p. 2) or, conversely, “dysfunctionaldispositions” (manual, p. 6) essentially corresponding to DSM-IV-TR personality disorders. Specificallyand respectively, the 11 HDS Scales paralleling dysfunctional DSM Behavioral Traits include: Excitable(Borderline Behavioral Traits), Skeptical (Paranoid), Cautious (Avoidant), Reserved (Schizoid), Leisurely(Passive-Aggressive; retained in the revised HDS but acknowledged by the test authors as being removedfrom the DSM), Bold (Narcissistic), Mischievous (Antisocial), Colorful (Histrionic), Imaginative(Schizotypal), Diligent (Obsessive-Compulsive), and Dutiful (Dependent). However, the test authorsprovide the following qualification regarding clinical application: “the HDS is not designed to measurepersonality disorders–the personality disorders are manifestations of mental disorder, and we are assessingself-defeating expressions of normal personality” (manual, pp. 6-7). The test authors go on to furtherclarify that “behavioral traits” are transient and contextually dependent; whereas “personality disorders”are “enduring and “pervasive.” Nevertheless, as the HDS is designed to be used for hiring and promotiondecisions (particularly for leadership positions), and thus constitutes a high stakes measure, users arecautioned to heed the authors’ cautionary statements regarding avoiding diagnostic or clinical applicationof the HDS, although human resource application of the HDS includes prescriptive professionaldevelopment information referred to as “Developmental Recommendations” (Leadership ForecastChallenge Report). These recommendations and applied interpretations of test results are in the form ofhelpful advice for improving interpersonal relationships and collaboration within the work environment.

The test booklet and manual administration instructions for the 168-item, dichotomously scored (i.e.,true/false) HDS are clear and easy to follow. A Flesch-Kincaid reading analysis for the HDS indicated it iswritten at a Grade 6.9 level. The manual includes administration instructions for accommodatingindividuals with disabilities, using the Online Internet System, using language translations of the HDS, andinformed consent. The test items are easy to comprehend, and a bubble format answer sheet is utilized,scored by the test publisher, with follow-up interpretive reports provided. However, online administration




is also available. The test requires 15 to 20 minutes to administer and is available in 42 languages,including English. Although ages are not specified, the manual states that the HDS is intended for use with“normal” adults, and not children, adolescents, or clinical/psychiatric populations. Test results reports areprovided for the test administrator and test-taker, providing percentile ranks for the 11 scales andinterpretive information available for making personnel selection decisions and for career advisement. TheHogan Development Survey Manual is well-written and organized, and includes information pertaining totheoretical foundation, test development, reliability, validity, interpretation, utilization, administration,and norms.

DEVELOPMENT

The test authors provide considerable information regarding the development of the HDS, starting with acomprehensive review of the relevant literature relating interpersonal competencies with leadership andorganizational dynamics. This is viewed from the standpoint of “performance risks” resulting fromdysfunctional dispositions arising out of the individual’s social development. From this perspective, allindividuals manifest these risks, but to different degrees. From an organizational perspective, this may beconceptualized as managerial competence.

The central construct of the HDS is interpersonal competency theoretically affecting organizational health.Eleven normatively distributed factors representing dimensions of interpersonal competency are identified.The 11 factors essentially correspond to DSM personality disorders, although the test authors emphasizethat the HDS is not intended to be a measure of pathology. Rather, the HDS normative scales collectivelyindicate “good interpersonal skills, to flawed skills, to non-existent skills” (manual, p. 8). Thus, a fewpeople are “highly effective” based on the scales; a few on the other end of the spectrum are consistentlyincompetent; and most individuals are somewhere in the middle. However, an individual may score highor low on any particular dimension or scale. The test authors cite several personality taxonomies–including Horney’s “neurotic needs,” Wiggins’s “interpersonal circumplex,” and Bentz’s “overridingpersonality defects”–as being the theoretical foundation of the HDS.

TECHNICAL

The test manual provides considerable technical information, with numerous tables summarizing dataaddressing the development, validation, and renorming of the HDS. Several studies validating the HDS arereviewed in detail, including meta-analysis results for scales aligned with competency domains. The finalnorming sample consisted of 109,103 participants, which included 46,135 managers/executives; andreflected demographics matching the U.S. Census Bureau’s American Fact Finder program (2006)–corresponding distribution tables are provided for age, gender, and ethnicity.




Reliability

Alpha coefficients ranged from .43 (Leisurely Scale) to .68 (Cautious and Colorful Scales). Test-retestreliability was conducted using retest intervals of less than 3 months and 9–12 months. Pearson correlationcoefficients, considered acceptable, ranged from .64 to .75 for the former and .52 to .75 for the latter, withregard to the 11 scales. Normalized Euclidean Similarities coefficients respectively ranged from .76 to .85and .75 to .85.

Validity

Several studies are reported investigating validity evidence in support of HDS score use. In considerationof construct validity evidence, HDS correlation coefficients were examined for various personality,interests, abilities, and preferences measures, including: the Hogan Personality Inventory (HPI), theCalifornia Psychological Inventory (CPI), the NEO PI-R, the International Personality Item Pool, theSixteen Personality Factor Questionnaire (16PF), the Motives, Values, Preferences Inventory (MVPI), theCampbell Interest and Skill Survey, the Jackson Personality Inventory, the Hogan Business ReasoningInventory, and the Watson-Glaser Critical Thinking Appraisal subtests. In general, the resulting coefficientswere in the directions predicted. Validity was further assessed through correlation of others’ descriptions,including ratings from observers, co-workers, spouses, and coaches–satisfactory levels of concordancewere observed. Finally, meta-analysis was used to examine the validity of test score interpretation for theHDS, using 26 independent samples (N = 3,059) from various professional publications between 1997and 2008. The meta-analysis specifically examined the relationship between personality and jobperformance competence. The results of the study were consistent with predicted outcomes.

COMMENTARY AND SUMMARY

The test authors provide a compelling review of the literature supporting the importance of interpersonalcompetencies as to their impact on organizations, particularly as this relates to individuals in leadershippositions. Indeed, for anyone who has worked in a complex organizational setting, the truth of this seemsobvious–all too often painfully obvious. Thus, the importance of the work invested by Robert Hogan andJoyce Hogan in developing the HDS and in conceptually framing leader effectiveness as it relates topersonality is monumental. Fortunately, the care that went into developing the HDS as a psychometricallyadequate and user-friendly tool for aiding personnel selection and professional growth lives up to theneed. The HDS manual is well-written and organized, as are the HDS Survey (test booklet), and spiral-bound reports (Select, Development, Lead). Human resource managers and selection committees shouldfind the HDS to be a very useful tool in objectively considering personality factors that potentially impactjob performance. Typically, this evaluation is more often done haphazardly through the interview process.




The HDS, perhaps in combination with performance measures and authentic assessments, holds thepromise of a more objective method of considering how personality factors impact interpersonal jobperformance.

This reviewer recommends the HDS as a tool to be used in combination with other sources of informationfor screening candidates seeking leadership positions. The HDS also promises utility as a staff developmenttool in targeting specific skill areas for enhancing interpersonal competence. The HDS meets acceptedpsychometric standards, and the test authors have done an excellent job of addressing technical features inthe respective published HDS test materials.

Cite this reviewAxford, S. (in press). [Test review of Hogan Development Survey [Revised]]. In J. F. Carlson, K. F. Geisinger, & J. L. Jonson (Eds.), Thenineteenth mental measurements yearbook. Retrieved from http://marketplace.unl.edu/buros/

REVIEW 2 OF 2

Review of the Hogan Development Survey [Revised] by THEODORE L.HAYES, Personnel Research Psychologist, U.S. Office of Personnel

Management, Washington, DC:

DESCRIPTION

The Hogan Development Survey (HDS) is a paper- or web-administered 168-item self-report measure of11 “dysfunctional dispositions … flawed interpersonal characteristics … themes that enhance and inhibiteffective performance” (manual, pp. 6, 9, & 11). The target respondent audience includes working-ageadults. Suggested applications include personnel selection, professional development or coaching,counseling, and research. The test authors state repeatedly that the HDS is not meant to diagnosepsychological disorders and should only be used in nonclinical situations. The HDS and test reports areavailable in 42 languages including English. Items, which are on average 9.8 words long and are pitched ata Grade 6.9 (13-year old) reading level, are administered using an Agree/Disagree response format. The11 scales each contain 14 items. The scales, which bear resemblance to personality disorders in theDiagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Text Revision (American PsychiatricAssociation, 2000), are labeled Excitable, Skeptical, Cautious, Reserved, Leisurely, Bold, Mischievous,Colorful, Imaginative, Diligent, and Dutiful. Graphical and interpretive reports are provided for the 11scale scores. There are no overall or otherwise aggregated scores.




DEVELOPMENT

The HDS is the result of work begun during the late 1980s to develop a test of dysfunctional dispositionsthat related empirically to nontest outcomes. The time was marked in behavioral science by the resurgenceof interest in developing measures of normal personality attributes known as the “Big Five” (Goldberg,1992; Hogan, 1991). Empirical research established that Big Five personality measures are taxonomicallyconsistent and linked to consequential work outcomes (e.g., Barrick & Mount, 1991). The HDS broadenedthe realm of psychometric personality assessment to include subclinical dysfunctional dispositions whilealso capitalizing on the groundswell of acceptance of personality measurement within organizations.Outside of the HDS, some research has attempted to align Big Five measures with counterproductivecriterion behavior (e.g., Salgado, 2002) or to create “syndrome scales” as part of omnibus personality testdevelopment (Schmit, Kihm, & Robie, 2000). Other competency- (van Velsor & Leslie, 1995) orpersonality-based (Brandon & Seldman, 2004) approaches exist that are more modest and atheoretical.Overall, the HDS is probably the only assessment of its kind.

TECHNICAL

Standardization

Scale score standardization and normative value setting for this edition of the HDS are executedexceptionally well. Over 109,000 complete cases were used to develop percentile ranks and expectedvalues for major demographic groups, typically interactions of groups (e.g., Table 6.14, “Norming samplescale means and standard deviations for ethnic groups by females under 40,” manual, p. 170). Singledemographic tables are provided for ethnicity, not age and gender. No score tables are presented byoccupational category. No scale score mean difference analyses, such as t-tests, are provided. Visualinspection indicates that most scale scores are consistent across demographic categories. A test user shouldhave confidence that these normative values are current and applicable, especially for professionaldevelopment or counseling purposes.

Reliability

The test publisher has chosen both test stability and internal consistency for the HDS reliability model. Thetest manual indicates that its test development approach considers external behavior more important thaninternal consistency. As a consequence, the alpha coefficients for the 11 scales range from .43 to .68(median = .61), which are notably low. Additionally, alpha coefficients are higher for 10 of the 11 scaleswhen data were gathered for professional development compared to data gathered for selection purposes.




A planned HDS revision incorporating homogeneous item composites (manual, p. 12) will remedy theinternal reliability issue. Stability estimates are presented for retesting within 3 months (average = .70)and between 9 to 12 months (average = .64). These values are within the range expected for theMinnesota Multiphasic Personality Inventory (MMPI; Parker, Hanson, & Hunsley, 1988) and somewhathigher than found for Big Five personality attributes (Roberts & Del Vecchio, 2000).

Validity

Content-oriented validity evidence for use of HDS test scores is highly sophisticated. Items were written toreflect dysfunctional criterion behaviors while also maintaining subtlety. Most items are written to reflectbehavior in a work setting. There is no item overlap between scales.

Conceptual or construct-level validity evidence supporting HDS test score use is the strongest suit. Theunderlying theories are rich and cannot be adequately described here. In brief, HDS scales reflect Horney’s(1950) tripartite model of flawed social interaction strategies. Young people develop interpersonal skillsand affective tendencies as results of beliefs and expectations regarding how others will treat them, leadingto adaptation of interpersonal schemas. Interpersonal deficiencies are based on flawed strategies used tomanage schema-based interaction expectations. These flawed strategies include moving away from people(Excitable, Skeptical, Cautious, Reserved, Leisurely), moving against them (Bold, Mischievous, Colorful,Imaginative), or moving toward them (Diligent, Dutiful). Higher HDS scale scores reflect flawed strategiesthat lead to workplace dysfunction. The test manual notes in passing that exceptionally low scores alsoreflect flawed strategies as if by dint of an extraordinary deficiency of any strategy. The test manualprovides developmental feedback ideas for each scale to correct flawed schemas.

Criterion-oriented validity evidence is more diffuse. The test authors are leaders in aligning criteria withpredictors using job analyses (e.g., Hogan & Holland, 2003). A large appendix in the test manual showsmeta-analyzed correlations using this approach combined with adjectival ratings, and other tables showsimilar adjectival ratings criteria arising from implementation of the HDS in a community survey. Thesedata indicate that self-reports from the HDS align with adjectival ratings made by others. Separately, theHDS seems to correlate with workplace multicourse ratings in theoretically consistent ways (Kaiser &Hogan, 2007; Kaiser, Le Breton, & Hogan, in press). At the same time, there is no body of correlationaldata linking the HDS and discrete organizational outcomes such as failure to promote, early termination,and so forth; even more so than with at-work theft, it may be exceptionally difficult to capture thesecriteria. It seems that criterion validity evidence for the HDS is more explanatory than predictive.

COMMENTARY

The HDS test manual is among the best this reviewer has ever reviewed for the Mental Measurements




Yearbook, and is clearly superior to its predecessors and competition. The undergirding theoretical basisfor the HDS is much stronger than for the Big Five.

The test publisher has positioned the HDS as being useful for multiple purposes, including pre-employment testing. Because the great majority of test-takers are likely to have some elevated score–andagain, low scores also are problematic–and because there is no apparent way to set a cutscore, it wouldseem the HDS is better suited for professional development than selection. A selection application mightbe appropriate in situations where the applicants’ test materials were first reviewed by trained consultantswho then could present the portfolio of strengths and limitations, “bright side” and “dark side,” for allcandidates so that a hiring manager or selection committee could be aware of the potential new hire’sdevelopmental challenges.

The approach taken in the HDS is that dysfunctional dispositions can derail individuals or teams at anylevel. Other approaches focus on derailment of high-potential leaders (e.g., Civil Service College ofSingapore, 2010; McCall & Lombardo, 1983). The focus on identified leaders is more operational and mayallow for easier criterion capture. Derailment seems to be a special case of dysfunction, though, and not alldysfunction is at the manager/leader level.

Although HDS scores have satisfactory empirical stability, an open question, and one not addressed in theHDS manual, is: how much behavioral improvement, not scale score change, would be expected for anindividual with dysfunctional tendencies in a professional development engagement? Optimisticapproaches to this question are offered by Davies (2009) and Gentry and Chappelow (2009), thoughcompelling empirical evidence is lacking. Smither, London, Flautt, Vargas, and Kucine (2003) found thatmanagers working with executive coaches after multisource feedback engaged in positive developmentalactivities, such as developing goals, and seemed to get improved ratings relative to managers not workingwith coaches. The answer to the question of whether and how much improvement occurs may depend onthe skill of the coach, motivation of the recipient, type of dysfunction, and type of change sought. Giventhis framework, change in criterion dysfunctional behavior should be associated with changes inHDS/criterion correlations even if the HDS scores themselves do not change over time.

Finally, the HDS materials and the feedback it provides are entirely adequate, especially when integratedwith the suite of tests offered by the test publisher. However, the score bar graph reports are both static-looking and oriented at single-scale feedback, compared to configural scale clusters based on the Horneyclassifications. Enhanced Web 2.0 reporting features might explore higher quality graphics, configuralfeedback, and possibly even interactive reporting that would allow clients to build personalized reports orgoal-setting plans.

SUMMARY




The HDS is alone in its test space and it has been developed with exceptional psychological andpsychometric care. Its proper place in an assessment battery may not yet be apparent. Unlike most tests,one does not review HDS feedback with a sense of personal esteem, mastery, or interpersonal competence.Maybe that is the point.

REVIEWER'S REFERENCES

American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (4th ed.,text rev.). Washington, DC: Author.

Barrick, M. R., & Mount, M. K. (1991). The Big-Five personality dimensions and job performance: A meta-analysis. Personnel Psychology, 44, 1-26.

Brandon, R., & Seldman, M. (2004). Survival of the savvy: High-integrity political tactics for career andcompany success. New York, NY: Free Press.

Civil Service College of Singapore. (2010). Research study: Understanding management derailment(technical report). Singapore: Centre for Leadership Development.

Davies, M. R. (2009). Unlocking the value of exceptional personalities. In R. B. Kaiser (Ed.), The perils ofaccentuating the positive (pp. 137-156). Tulsa, OK: HoganPress.

Gentry, W. A., & Chappelow, C. T. (2009). Managerial derailment: Weaknesses that can be fixed. In R. B.Kaiser (Ed.), The perils of accentuating the positive (pp. 99-113). Tulsa, OK: HoganPress.

Goldberg, L. R. (1992). The development of markers for the Big Five factor structure. PsychologicalAssessment, 4, 26-42.

Hogan, J., & Holland, B. (2003). Using theory to evaluate personality and job-performance relations: Asocioanalytic perspective. Journal of Applied Psychology, 88, 100-112.

Hogan, R. (1991). Personality and personality measurement. In M. D. Dunnette & L. M. Hough (Eds.),Handbook of industrial and organizational psychology (vol. 2, 2nd ed., pp. 327-396). Palo Alto, CA:Consulting Psychologists Press.

Horney, K. (1950). Neurosis and human growth. New York, NY: Norton.

Kaiser, R. B., & Hogan, R. (2007). The dark side of discretion: Leader personality and organizationaldecline. In R. Hooijberg, J. Hunt, J. Antonakis, & K. Boal (Eds.), Being there even when you are not:

10All Mental Measurements Yearbook test reviews are copyrighted by the Buros Center for Testing. Reviews may be printed for individual useonly, and may not be otherwise duplicated or distributed without written permission. Please refer to the Buros website for further informationabout copyright and permissions: http://www.buros.org/copyright-and-permissions.



Leading through strategy, systems, and structures. Monographs in Leadership and Management, vol. 4 (pp.177-197). London, England: Elsevier Science.

Kaiser, R. B., LeBreton, J. M., & Hogan, J. (in press). The dark side of personality and ineffectiveleadership. Applied Psychology: An International Review.

McCall, M. W., & Lombardo, M. M. (1983). Off the track: Why and how successful executives get derailed.Greensboro, NC: Centre for Creative Leadership.

Parker, K. H., Hanson, R. K., & Hunsley, J. (1988). MMPI, Rorshach, and WAIS: A meta-analyticcomparison of reliability, stability, and validity. Psychological Bulletin, 103, 367-373.

Roberts, B. W., & Del Vecchio, W. F. (2000). The rank-order consistency of personality traits fromchildhood to old age: A quantitative review of longitudinal studies. Psychological Bulletin, 126, 3-25.

Salgado, J. F. (2002). The Big Five personality dimensions and counterproductive behaviors. InternationalJournal of Selection and Assessment, 10, 117-125.

Schmit, M. J., Kihm, J. A., & Robie, C. (2000). Development of a global measure of personality. PersonnelPsychology, 53, 153-193.

Smither, J. W., London, M., Flautt, R., Vargas, Y., & Kucine, I. (2003). Can working with an executive coachimprove multisource feedback ratings over time? A quasi-experimental field study. Personnel Psychology,56, 23-44.

Van Velsor, E., & Leslie, J. B. (1995). Why executives derail: Perspectives across time and cultures.Academy of Management Executive, 9, 62-72.

Cite this reviewHayes, T. L. (in press). [Test review of Hogan Development Survey [Revised]]. In J. F. Carlson, K. F. Geisinger, & J. L. Jonson (Eds.), Thenineteenth mental measurements yearbook. Retrieved from http://marketplace.unl.edu/buros/

11All Mental Measurements Yearbook test reviews are copyrighted by the Buros Center for Testing. Reviews may be printed for individual useonly, and may not be otherwise duplicated or distributed without written permission. Please refer to the Buros website for further informationabout copyright and permissions: http://www.buros.org/copyright-and-permissions.



test reviews - mentismentisglobal.com/wp-content/uploads/hogan_secure... · the test authors...

Documents