1 welcome!. 2 critical integrative metatheory: new methods to evaluate psychological theories &...
TRANSCRIPT
-
WELCOME!
-
Critical Integrative Metatheory: New Methods to Evaluate Psychological Theories & Models for Review Steven E. Wallis, PhD [email protected]
Director, Foundation for the Advancement of Social TheoryAdjunct Faculty, Capella UniversityFellow, Institute for Social Innovation, Fielding Graduate UniversityEditorial Board, Integral Review
International Congress of Psychology 2012July 22-27 Cape Town, South Africa
-
We want psychology to be of greater benefit to humanityLearningPsychoanalysisMotivationCommunicationBehaviorismCognitionStressEmotions
-
Emergence of PsychologyPsychology is NOT much appreciated by other sciences or the general public. For GOOD reasons!We are not advancing as a science there is no proof that psychology is better now than 50 years ago. Why have we failed to improve?
-
One Path to Improving Psychology is the Process of Submission & ReviewReviewSubmissionTheory Creation
-
Many Rigorous Standards for Evaluating MethodsParticipant CharacteristicsSampling ProceduresSampling size, power, precisionMeasures and CovariatesResearch DesignRecruitment StatisticsData AnalysisEtc., etc., etc., etc., etc.,
Methods Review
-
Standards for Evaluating Theory? TheoryReview
-
What is theory?LensMapMetaphorStoryDiagramNarrativeSet of axiomsModelEthicsPolicyMental modelSchemaMind mapAssumptionsThese are all names for a conceptual construct that may be useful for engaging the world.
-
TheoryA theory is a conceptual construct, a set of abstract causal relationships.Empowers (to take effective action)Restricts (limits our sense of truth)Is developed by education, experience, and social construction
Is (apparently) INNESCAPABLE(so it is apparently IMPORTANT)
-
MetaphoricallyMetatheory is a theoretical lens that may be directed toward the investigation of theory.
-
Or, more formallyMetatheory is primarily the study of theory, including the development of overarching combinations of theory, as well as the development and application of theorems for analyses that reveal underlying assumptions about theory and theorizing.
-
Three Ways to Evaluate TheoryCreation
Critical AnalysisCategorization
-
Common Methods for Evaluating the Creation of Theory (and issues)Voracious reading (what? How much?)Be brave, Creative, etc (how to measure?)Research (empirical data what is data?)Synthesize existing theory (recursive problem)Reflexivity (how to measure?)Abstraction (How much is best?)Grounded Theory (more structured approach)Reflexive Dimensional Analysis (more structured approach)
-
Common Methods for Categorization of TheoryHistoricalGeographicalUnit level / Middle Range / GrandStyle (Literary, Academic, Eristic, etc.)Purpose (Analyzing, Explaining, Predicting)(ISSUE: Within a category, What Theories are Better?)
-
Common Methods for Critical Analysis of TheoryPlausibility ParsimonyCorrespondence to observed factsCoherence to existing theoryApplicationPropositional AnalysisFalsification(ISSUE: Rarely Done, Rarely Quantified)
-
CRITICAL Metatheory
Reflects The Need For RIGORBetter to have a specific metatheoretical lensThe results must be quantified
If our lens of metatheory is cloudy, we wont see the lens of theory with clarity
-
EXAMPLE: Propositional AnalysisIdentify the logical propositions.Diagram the causal relationships.Integrate diagrams.Identify and count the Concatenated aspects (two or more causal influences).Count the total number of aspects (Complexity of the theory).Calculate the Robustness(divide Concatenated aspects by total aspects).
-
Abstract ExampleADCBACCEPropositions within a TheoryCarefully IntegratedABCDEC = 5R = 0.20
-
Previous ResearchEvolution of Theory & Changes in ComplexityComplex theories some usefulness
Chart1
7
6
14
11
3
Total number of Aspects
Number of Aspects
Changes in the Complexity of Theory as it Evolves Through A Scientific Revolution Toward Greater Usefulness
Sheet1
YearTotal Number of AspectsNumber of Concatenated AspectsRobustness
100710.14Plutarch
1550610.17Cardan
16001430.21Gilbert
17501140.36Two fluid theory
1785331Coulomb
YearRobustness
1000.14
15500.17
16000.21
17500.36
17851
Sheet1
00
00
00
00
00
Robustness
Year
Robustness
Theories of Electrostatic Attraction
Sheet2
00
00
00
00
00
Total number of Aspects
Number of Concatenated Aspects
Year
Number of Aspects
Change in Aspects Over Time
Sheet3
-
Previous ResearchEvolution of Theory & Changes in RobustnessRobust theories Very Useful
Chart2
0.140.14
0.170.17
0.210.21
0.360.36
11
Robustness
Year
Robustness
Changes in Robustness of Theory as it Evolves Through A Scientific Revolution Toward Greater Usefulness
Sheet1
YearTotal Number of AspectsNumber of Concatenated AspectsRobustness
100710.14Plutarch
1550610.17Cardan
16001430.21Gilbert
17501140.36Two fluid theory
1785331Coulomb
YearRobustness
1000.14
15500.17
16000.21
17500.36
17851
Sheet1
00
00
00
00
00
Robustness
Year
Robustness
Theories of Electrostatic Attraction
Sheet2
00
00
00
00
00
Total number of Aspects
Number of Concatenated Aspects
Year
Number of Aspects
Change in Aspects Over Time
Sheet3
-
Theory of motivation and competence (White, 1959) 1. More new stimulation4. More effect on environment3. More action2. More cognition6. More feelings of efficacy (motivation)C = 6R = 0.17
-
Theory of planned behavior (Perkins et al., 2007) 1. More expected value2. More subjective norms3. More perceived behavioral control4. More Behavioral intentions5. More behaviorC = 5R = 0.20
-
Combined TheoriesC = 8R = 0.25
-
Comparing Psychological TheoriesMEASURABLE IMPROVEMENT
-
Approach Provides Editors and Reviewers With a Standard for Evaluating TheoryPropositional Analysis to determine Complexity and Robustness Theories of higher Complexity are privileged Theories of higher Robustness are privilegedTheoryReview
-
To Improve, We Must Push Theories of Psychology to Higher Levels of Complexity and Robustness
-
Without Rigorous Standards We Become Biased and Our Theories Become SimpleTheory of motivation
Change in Complexity
90 years of (de) evolution
150 years until it vanishes?!?
Chart1
10
8
13
6
6
7
10
6
7
Complexity
Sheet1
AuthorYearComplexityRobustnessNumber of citationsDecadeNumber of articles within DecadeList of all AspectsNOTESClarity of WritingOrigins of theoryYearAtomistic LogicsLinear LogicsCircular LogicsBranching LogicsConcatenated LogicsNumber of DiagramsComplexityRobustness multiplied by 10Use of Diagrams?
Perrin1923100111921-19303Fuzzy languagesynthesizes three extant theories1923130100100No
Mowrer1938801061931-19404Fuzzy languagesome experimental results have led to rethinking of theory193803010080No
Maslow194313070181941-19505Maslow may be highly cited because the high number of atomistic statements in his theory allow/invite others to interpret and link the atomistic logics into more complex ones.Extant theory, some experimental influence, mostly from clinical experience1943420200130No
White195960.1752031951-19607Discontent with extant theories of motivation based on drives - striving to find new insights through synthesis of multiple trends of thought1959011010617No
Valenstein, Cox, & Kakolewski1970602471961-19709Based on experiment197005000060No
Solomon & Corbit197470.299991971-19805Diagram is "illustrative" different from textEmprical and observational of daily events1974010221729Yes
Weiner1985130.0863371981-19905Detail LevelAbove averageSynthesis of multiple theories1985030221014.5Yes
Weiner198570.2163371981-19905High LevelAbove averageSynthesis of multiple theories199100010060Yes
Markus & Kitayama19916086391991-20002Markus & Kitayama - similar to maslow have a model that is more open to interpretation - so many cites (BAD for advancing the science, good for advancing personal careers - the selfish scientistAbove averageInterdisciplinary - connecting psychology with (cultural) anthropology2004030012721No
Baker, et al.200460.174792000-20103Figure 1Quite clearGeneralizations of previous studies and theoriesYes
Baker, et al.200480.254792000-20103Figure 2Quite clearGeneralizations of previous studies and theoriesYes
Worst clarity is from complex, exception-ridden, run-on discussions
Best clarity is from clear hypotheses/propositions
ROBUSTNESS TIMES 10
YearComplexityRobustness
1923100
193880
1943130
195960.17
197060
197470.29
1985100.145
199160
200470.21
WITH AVERAGED VALUESWITH AVERAGED VALUES
YearComplexityRobustnessYearComplexityRobustness
19231001923100
193880193880
19431301943130
195960.17195960.17
197060197060
197470.29197470.29
average1985100.1451985100.145
199160199160
average200470.021200470.021
10070.1410070.14
155060.17155060.17
1600140.211600140.21
1750110.361750110.36
178531178531
Because I'm looking for the C&R of each paper's contribution to the theoretical literature, I averaged the C&R when two models were presented (Baker, et al) and when two interpretations of a model seemed required ( Weiner's model could be viewed at high lBecause I'm looking for the C&R of each paper's contribution to the theoretical literature, I averaged the C&R when two models were presented (Baker, et al) and when two interpretations of a model seemed required ( Weiner's model could be viewed at high l
For Robustness, there does not appear to be a meaningful trend.For Robustness, there does not appear to be a meaningful trend.
For Complexity, there appears to be a trend toward simplicity.For Complexity, there appears to be a trend toward simplicity.
We are not merely wandering, but appear to be following the guide of parsimony - a guide whose treustworthiness is gravely in doubtWe are not merely wandering, but appear to be following the guide of parsimony - a guide whose treustworthiness is gravely in doubt
Meehl, 2002 p. 345 "they only one [criteria of theory appraisal] they all mention (except Poperians) is parsimony"Meehl, 2002 p. 345 "they only one [criteria of theory appraisal] they all mention (except Poperians) is parsimony"
This is a mere 100 years compared with the thousands of the previous studyThis is a mere 100 years compared with the thousands of the previous study
KEY - Meehl makes suggestions to determine which theories are best based (in part) on application. But HOW do we choose which ones to apply?KEY - Meehl makes suggestions to determine which theories are best based (in part) on application. But HOW do we choose which ones to apply?
AND - that only tells us which ones are best AFTER we already know it works. SO what I'm doing is saying which direction shoud we go to find those theories that work!AND - that only tells us which ones are best AFTER we already know it works. SO what I'm doing is saying which direction shoud we go to find those theories that work!
You must avoid the tank-trap of parsimony and climb the difficult mountain of complexity before you reach the fertile vale of RobustnessYou must avoid the tank-trap of parsimony and climb the difficult mountain of complexity before you reach the fertile vale of Robustness
Sheet1
00
00
00
00
00
00
00
00
00
Complexity
Robustness
Year
Complexity and Robustness of Psychological Theories Over Time
Sheet2
00
00
00
00
00
00
00
00
00
Complexity
Robustness
Sheet3
000000000
000000000
000000000
000000000
000000000
000000000
000000000
000000000
000000000
Atomistic Logics
Linear Logics
Circular Logics
Branching Logics
Concatenated Logics
Number of Diagrams
Complexity
Robustness multiplied by 10
0
0
0
0
0
0
0
0
0
Complexity
MBD009182BC.xls
Chart2
0.140.14
0.170.17
0.210.21
0.360.36
11
Robustness
Year
Robustness
Theories of Electrostatic Attraction
Sheet1
YearTotal Number of AspectsNumber of Concatenated AspectsRobustness
100710.14Plutarch
1550610.17Cardan
16001430.21Gilbert
17501140.36Two fluid theory
1785331Coulomb
YearRobustness
1000.14
15500.17
16000.21
17500.36
17851
Sheet1
00
00
00
00
00
Robustness
Year
Robustness
Theories of Electrostatic Attraction
Sheet2
00
00
00
00
00
Total number of Aspects
Number of Concatenated Aspects
Year
Number of Aspects
Change in Aspects Over Time
Sheet3
MBD0091876A.xls
Chart1
71
61
143
114
33
Total number of Aspects
Number of Concatenated Aspects
Year
Number of Aspects
Change in Aspects Over Time
Sheet1
YearTotal Number of AspectsNumber of Concatenated AspectsRobustness
100710.14Plutarch
1550610.17Cardan
16001430.21Gilbert
17501140.36Two fluid theory
1785331Coulomb
YearRobustness
1000.14
15500.17
16000.21
17500.36
17851
Sheet1
00
00
00
00
00
Robustness
Year
Robustness
Theories of Electrostatic Attraction
Sheet2
00
00
00
00
00
Total number of Aspects
Number of Concatenated Aspects
Year
Number of Aspects
Change in Aspects Over Time
Sheet3
-
Without Rigorous Standards We Become Biased and Our Theories Become WeakTheory of motivation
Change in Robustness
90 years of (weak) evolution
900 years to success?!?
Chart2
0
0
0
0.17
0
0.29
0.145
0
0.021
Robustness
Year
Sheet1
AuthorYearComplexityRobustnessNumber of citationsDecadeNumber of articles within DecadeList of all AspectsNOTESClarity of WritingOrigins of theoryYearAtomistic LogicsLinear LogicsCircular LogicsBranching LogicsConcatenated LogicsNumber of DiagramsComplexityRobustness multiplied by 10Use of Diagrams?
Perrin1923100111921-19303Fuzzy languagesynthesizes three extant theories1923130100100No
Mowrer1938801061931-19404Fuzzy languagesome experimental results have led to rethinking of theory193803010080No
Maslow194313070181941-19505Maslow may be highly cited because the high number of atomistic statements in his theory allow/invite others to interpret and link the atomistic logics into more complex ones.Extant theory, some experimental influence, mostly from clinical experience1943420200130No
White195960.1752031951-19607Discontent with extant theories of motivation based on drives - striving to find new insights through synthesis of multiple trends of thought1959011010617No
Valenstein, Cox, & Kakolewski1970602471961-19709Based on experiment197005000060No
Solomon & Corbit197470.299991971-19805Diagram is "illustrative" different from textEmprical and observational of daily events1974010221729Yes
Weiner1985130.0863371981-19905Detail LevelAbove averageSynthesis of multiple theories1985030221014.5Yes
Weiner198570.2163371981-19905High LevelAbove averageSynthesis of multiple theories199100010060Yes
Markus & Kitayama19916086391991-20002Markus & Kitayama - similar to maslow have a model that is more open to interpretation - so many cites (BAD for advancing the science, good for advancing personal careers - the selfish scientistAbove averageInterdisciplinary - connecting psychology with (cultural) anthropology2004030012721No
Baker, et al.200460.174792000-20103Figure 1Quite clearGeneralizations of previous studies and theoriesYes
Baker, et al.200480.254792000-20103Figure 2Quite clearGeneralizations of previous studies and theoriesYes
Worst clarity is from complex, exception-ridden, run-on discussions
Best clarity is from clear hypotheses/propositions
ROBUSTNESS TIMES 10
YearComplexityRobustness
1923100
193880
1943130
195960.17
197060
197470.29
1985100.145
199160
200470.21
WITH AVERAGED VALUESWITH AVERAGED VALUES
YearComplexityRobustnessYearComplexityRobustness
19231001923100
193880193880
19431301943130
195960.17195960.17
197060197060
197470.29197470.29
average1985100.1451985100.145
199160199160
average200470.021200470.021
10070.1410070.14
155060.17155060.17
1600140.211600140.21
1750110.361750110.36
178531178531
Because I'm looking for the C&R of each paper's contribution to the theoretical literature, I averaged the C&R when two models were presented (Baker, et al) and when two interpretations of a model seemed required ( Weiner's model could be viewed at high lBecause I'm looking for the C&R of each paper's contribution to the theoretical literature, I averaged the C&R when two models were presented (Baker, et al) and when two interpretations of a model seemed required ( Weiner's model could be viewed at high l
For Robustness, there does not appear to be a meaningful trend.For Robustness, there does not appear to be a meaningful trend.
For Complexity, there appears to be a trend toward simplicity.For Complexity, there appears to be a trend toward simplicity.
We are not merely wandering, but appear to be following the guide of parsimony - a guide whose treustworthiness is gravely in doubtWe are not merely wandering, but appear to be following the guide of parsimony - a guide whose treustworthiness is gravely in doubt
Meehl, 2002 p. 345 "they only one [criteria of theory appraisal] they all mention (except Poperians) is parsimony"Meehl, 2002 p. 345 "they only one [criteria of theory appraisal] they all mention (except Poperians) is parsimony"
This is a mere 100 years compared with the thousands of the previous studyThis is a mere 100 years compared with the thousands of the previous study
KEY - Meehl makes suggestions to determine which theories are best based (in part) on application. But HOW do we choose which ones to apply?KEY - Meehl makes suggestions to determine which theories are best based (in part) on application. But HOW do we choose which ones to apply?
AND - that only tells us which ones are best AFTER we already know it works. SO what I'm doing is saying which direction shoud we go to find those theories that work!AND - that only tells us which ones are best AFTER we already know it works. SO what I'm doing is saying which direction shoud we go to find those theories that work!
You must avoid the tank-trap of parsimony and climb the difficult mountain of complexity before you reach the fertile vale of RobustnessYou must avoid the tank-trap of parsimony and climb the difficult mountain of complexity before you reach the fertile vale of Robustness
Sheet1
0
0
0
0
0
0
0
0
0
Robustness
Year
Complexity and Robustness of Psychological Theories Over Time
Sheet2
00
00
00
00
00
00
00
00
00
Complexity
Robustness
Sheet3
000000000
000000000
000000000
000000000
000000000
000000000
000000000
000000000
000000000
Atomistic Logics
Linear Logics
Circular Logics
Branching Logics
Concatenated Logics
Number of Diagrams
Complexity
Robustness multiplied by 10
0
0
0
0
0
0
0
0
0
Complexity
MBD009182BC.xls
Chart2
0.140.14
0.170.17
0.210.21
0.360.36
11
Robustness
Year
Robustness
Theories of Electrostatic Attraction
Sheet1
YearTotal Number of AspectsNumber of Concatenated AspectsRobustness
100710.14Plutarch
1550610.17Cardan
16001430.21Gilbert
17501140.36Two fluid theory
1785331Coulomb
YearRobustness
1000.14
15500.17
16000.21
17500.36
17851
Sheet1
00
00
00
00
00
Robustness
Year
Robustness
Theories of Electrostatic Attraction
Sheet2
00
00
00
00
00
Total number of Aspects
Number of Concatenated Aspects
Year
Number of Aspects
Change in Aspects Over Time
Sheet3
MBD0091876A.xls
Chart1
71
61
143
114
33
Total number of Aspects
Number of Concatenated Aspects
Year
Number of Aspects
Change in Aspects Over Time
Sheet1
YearTotal Number of AspectsNumber of Concatenated AspectsRobustness
100710.14Plutarch
1550610.17Cardan
16001430.21Gilbert
17501140.36Two fluid theory
1785331Coulomb
YearRobustness
1000.14
15500.17
16000.21
17500.36
17851
Sheet1
00
00
00
00
00
Robustness
Year
Robustness
Theories of Electrostatic Attraction
Sheet2
00
00
00
00
00
Total number of Aspects
Number of Concatenated Aspects
Year
Number of Aspects
Change in Aspects Over Time
Sheet3
-
To ConcludeEmpirical data is theory laden we cannot understand data without understanding theory.If we are make psychology more beneficial to humanity we must directly address theory.If we are to have effective theory, that theory must be addressed in a rigorous scientific way (Critical Integrative Metatheory)Those methods must be applied to review submissions to journals
-
Many Thanks!Steven E. Wallis, Ph.D.
Director, Foundation for the Advancement of Social Theory http://projectfast.org Fellow, Institute for Social Innovation, Fielding Graduate UniversityAdjunct Faculty, Capella University Editorial Board, Integral Review
[email protected] Graduate University is a 501 c (3) non-profit university of higher learning.
Source: Wallis (building on work of Dubin, Kaplan, Stinchcombe)Explanation: Based on causal relationships and concatenation. Here are four components, one is concatenated (resultant from two or more causal components) So, Robustness (measure of structure) is one divided by four = 0.25. Strength: Repeatable, provides objective path for advancing theory. Robustness has been related to efficacy in practical application.Limit: Based on ideas of integral thinking and complexity theory which have not themselves been proved