Empirical Methods – G. Grote ETHZ, Fall08
Obectives of the course
• Basic knowledge of social science methods (design, data collection and analysis)
• Ability to assess quality of empirical research• Ability to design empirical studies in an
organizational context• Ability to evaluate/hire external experts for
carrying out empirical studies in organi-zations
Empirical Methods – G. Grote ETHZ, Fall08
Examples of social science methods used in organizational contexts• Interview
– Expert interview
– Performance appraisal
• Questionnaire– Customer survey
– Personality test
• Observation– Communication analysis
– Shadowing
• Archival data– Meeting protocols
– Personnel data
Empirical Methods – G. Grote ETHZ, Fall08
Special issues in social research
• Control of complexity– Constraints on manipulation of study conditions– Studying "hypothetical constructs"– Limited possibilities for reducing complexity
• Studies with humans– Effects through researcher / researched individual and social
embeddedness– Ethical principles: voluntary participation, informed consent, no
physical/psychological harm, confidentiality
It is all about grey zones, there is no black and white Building competence for navigating in these grey zones.
Empirical Methods – G. Grote ETHZ, Fall08
Special issues for social research in organizational contexts
• Objectives for study by different stakeholders in the organization
• Role of people carrying out the study
• Role of people being studied
• Distribution of control over use of results
Empirical Methods – G. Grote ETHZ, Fall08
Empirical methods: Overview
Top ic Tutorial 1 Tutorial 2 Tutorial 3 Lecturer
17.9. Introduction Grote
24.9. Study design Grote
1.10. Definition of variables Grote
8.10. Methods for data collection and analysis Grote
15.10. Questionnaire: Design of questions and response formats Belting
22.10. Data analysis: Types of data and statistical tests Belting
29.10. Interviews: Overview Belting
5.11. Example: Expert interview Belting
12.11. Guest speaker Belting
19.11. Observation: Overview Kolbe
26.11. Observation: Data collection and analysis Kolbe
3.12. Data collection for tutorial 3: Non-technical skills for team coordination Kolbe
10.12. Example: Interaction in high-risk teams Kolbe
17.12. Preparation for exam All
Empirical Methods – G. Grote ETHZ, Fall08
Textbook
• Darren Langdridge, Introduction to Research Methods and Data Analysis in Psychology. Pearson Prentice-Hall, 2004. ISBN 0-13-097832-9
Empirical Methods – G. Grote ETHZ, Fall08
Organization of course
• 3 ETCS points (approx. 75-90 work hours).• Besides the lecture, the prerequisite for credits
points and exam participation is the completion of three tutorials.
• Tutorials are worked on in groups of 2-4 students, there is one tutorial per course part (study design; survey&quantitative analysis; observation)
• The exam is written and open book; provisional date: Jan. 7, 2009, 13:15-14:45. Overall grade: 50% tutorials & 50% exam.
• Material for the lecture at www.oat.ethz.ch.
Empirical Methods – G. Grote ETHZ, Fall08
Empirical methods: Overview
Top ic Tutorial 1 Tutorial 2 Tutorial 3 Lecturer
17.9. Introduction Grote
24.9. Study design Grote
1.10. Definition of variables Grote
8.10. Methods for data collection and analysis Grote
15.10. Questionnaire: Design of questions and response formats Belting
22.10. Data analysis: Types of data and statistical tests Belting
29.10. Interviews: Overview Belting
5.11. Example: Expert interview Belting
12.11. Guest speaker Belting
19.11. Observation: Overview Kolbe
26.11. Observation: Data collection and analysis Kolbe
3.12. Data collection for tutorial 3: Non-technical skills for team coordination Kolbe
10.12. Example: Interaction in high-risk teams Kolbe
17.12. Preparation for exam All
Empirical Methods – G. Grote ETHZ, Fall08
Basic research design
Independent variable/sConditions/Causes
Dependent variable/sEffects
Intervening variable/sConfounding variable/s
Empirical Methods – G. Grote ETHZ, Fall08
Example
• How are innovation, technology use and firm performance related?– What is cause and effect?– What are important intervening variables?
Empirical Methods – G. Grote ETHZ, Fall08
Steps in empirical research
• Define general question of interest• Draw on existing knowledge to develop
specific questions/hypotheses to be studied
• Determine design of study• Data collection• Data analysis• Interpretation/conclusions regarding initial
questions/hypotheses
Empirical Methods – G. Grote ETHZ, Fall08
Developing a study design (1)
• Deciding on overall objective:– Testing hypotheses (incl. evaluation)– Generating hypotheses– Describing a population– Testing methods
• Defining variables of interest– Analytical definition– Operational definition
determining methods for data collection
Empirical Methods – G. Grote ETHZ, Fall08
Criteria for good data collection methods
• Reliability: stability of measurement– Internal consistency– Retest– Split half
• Validity: measuring what is intended to be measured– Face validity– Criterion validity: concurrent, predictive– Construct validity
Empirical Methods – G. Grote ETHZ, Fall08
Developing a study design (2)
• Determine sample – Type of sample: Random, stratified,
cluster, convenience, etc.– Size of sample– Case study
• Determine methods for data analysis
• Planning resources
• Internal and external validity check
Empirical Methods – G. Grote ETHZ, Fall08
Criteria for good study design• Internal validity = unambiguous interpretation of
results/few alternative explanations for results– Possible problems: Uncontrolled influences by e.g.
external events, selection of participants, repeated measuring.
• External validity = generalizability of results regarding other samples of people, situations, time– Possible problems: e.g. special character-istics of
participants, reactive measurement
Empirical Methods – G. Grote ETHZ, Fall08
Reporting social science research• Overall objectives
– Consistent, transparent documentation of research process– Building trust in interpretation/conclusions
• Structure of report– Management summary– Introduction/rationale for study– Motivation for specific questions studied based on existing
knowledge– Methods: sample, data collection and analysis– Results– Interpretation/Conclusions– Appendix: Instruments used, raw data etc.
Empirical Methods – G. Grote ETHZ, Fall08
Tutorial 1
• Please read the article and write a short critique (ca. 2-3 pages) of the article using the steps of study design, validity criteria and reporting guidelines presented in the lecture.
• You may work on this tutorial alone or together with another student.
• The article:Brown, S.L. & Eisenhardt, K.M. (1997). The art of continuous change: linking complexity theory and time-paced evolution in relentlessly shifting organizations. Administrative Science Quarterly, 42, 1-34.
• Deadline: Oct. 6, noon, hand in to [email protected]
Empirical Methods – G. Grote ETHZ, Fall08
Empirical methods: Overview
Top ic Tutorial 1 Tutorial 2 Tutorial 3 Lecturer
17.9. Introduction Grote
24.9. Study design Grote
1.10. Definition of variables Grote
8.10. Methods for data collection and analysis Grote
15.10. Questionnaire: Design of questions and response formats Belting
22.10. Data analysis: Types of data and statistical tests Belting
29.10. Interviews: Overview Belting
5.11. Example: Expert interview Belting
12.11. Guest speaker Belting
19.11. Observation: Overview Kolbe
26.11. Observation: Data collection and analysis Kolbe
3.12. Data collection for tutorial 3: Non-technical skills for team coordination Kolbe
10.12. Example: Interaction in high-risk teams Kolbe
17.12. Preparation for exam All
Empirical Methods – G. Grote ETHZ, Fall08
Types of variables
• Role in study design– independent– dependent– intervening
• Level of measurement: What do the numbers mean that we assign to a variable?– discrete variables: nominal– continuous variables: rank, interval, ratio
Empirical Methods – G. Grote ETHZ, Fall08
Identify different types of variables:
• "Larger firms are more innovative."• "Larger firms are more innovative in growth
markets."• "In open plane offices compared to cell offices
frequency of communication increases and duration decreases."
• "Higher levels of task uncertainty demand more personal forms of coordination."
• "Successful strategy implementation can only be top-down in stable markets."
Empirical Methods – G. Grote ETHZ, Fall08
Analytic and operational definition of variables
• Analytic definition:
Content/meaning of variable
• Operational definition:
Measurement of variable
Empirical Methods – G. Grote ETHZ, Fall08
Exercise on defining variables
• Are part-time employees more discriminated against in smaller companies?
• Does communication technology improve performance of software development teams?
• Does liberalization of energy markets decrease energy prices?
Which are the independent, dependent, and inter-vening variables? Provide analytic and operational definitions for all variables. What level of measure-ment do you suggest for each variable?
Empirical Methods – G. Grote ETHZ, Fall08
Empirical methods: Overview
Top ic Tutorial 1 Tutorial 2 Tutorial 3 Lecturer
17.9. Introduction Grote
24.9. Study design Grote
1.10. Definition of variables Grote
8.10. Methods for data collection and analysis Grote
15.10. Questionnaire: Design of questions and response formats Belting
22.10. Data analysis: Types of data and statistical tests Belting
29.10. Interviews: Overview Belting
5.11. Example: Expert interview Belting
12.11. Guest speaker Belting
19.11. Observation: Overview Kolbe
26.11. Observation: Data collection and analysis Kolbe
3.12. Data collection for tutorial 3: Non-technical skills for team coordination Kolbe
10.12. Example: Interaction in high-risk teams Kolbe
17.12. Preparation for exam All
Empirical Methods – G. Grote ETHZ, Fall08
Quantitative versus qualitative approaches to social science research
• Quantitative approaches: measuring and counting phenomena+ precise, controlled, claims on causation/prediction
- oversimplification, no recognition of subjectivity/ individuality
• Qualitative approaches: capturing qualities of phenomena+ recognition of subjectivity/individuality, openness
- difficult to establish internal and external validity, difficult to separate data analysis and interpretation
Empirical Methods – G. Grote ETHZ, Fall08
The seminar ... (from ASQ, 1979)
• Qualitative researcher: "Many people these days are bored with their work and are ..."
• Quantitative researcher (interrupting): "What people, how many, when do they feel this way, where do they work, what do they do, why are they bored, how long have they felt this way, what are their needs, when do they feel excited, where did they come from, what parts of their work bother them most, which ..."
• Qualitative researcher: "Never mind."
Empirical Methods – G. Grote ETHZ, Fall08
Overview of methods for data collection
• Archival data+ non-reactive, "condensed organizational knowledge"
- not aligned with purpose of the investigation
• Questionnaire + objective, applicable for large samples
- no control over the actual data collection, response biases
• Interview + control over data collection, treatment of complex issues possible
- resource-intensive, interviewer influence
• Observation + access to implicit knowledge, natural situation
- subjective meaning of the observed unknown, no control over the occurrence of the events under study
Empirical Methods – G. Grote ETHZ, Fall08
Examples: How the questions shape the answers (1)
• "Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened?" 33% responded with "possible"
versus
"Do you believe the Holocaust happened?" 89% yes, 3.5% no, 7% don´t know
• Taboo behaviors: don´t ask if but when the behavior has been shown in the past
Empirical Methods – G. Grote ETHZ, Fall08
Examples: How the questions shape the answers (2)
Own behavior yes/no
Percent yes
Own behavior frequency scale 1-5
Percent 4/5
Correct behavior yes/no
Percent yes
Correct behavior scale 1-5
Percent 4/5 Avoid use of spray cans 84 11 72 83 Bio-shopping during the last fortnight
16 9 96 41
Separate glass collection 96 93 100 89 Separate paper collection 94 95 98 85 Separate aluminium collection
78 62 90 69
Use compost 64 38 100 83 Collect batteries 92 98 100 91 Return unused drugs to pharmacy
32 33 91 69
Reduce water when showering/taking a bath
40 31 66 63
Empirical Methods – G. Grote ETHZ, Fall08
Examples: How the questions shape the answers (3)
Zur Anzeige wird der QuickTime™ Dekompressor „TIFF (Unkomprimiert)“
benötigt.
Zur Anzeige wird der QuickTime™ Dekompressor „TIFF (Unkomprimiert)“
benötigt.
Empirical Methods – G. Grote ETHZ, Fall08
Overview of methods for data analysis
• Overall objective of data analysis– Description– Reduction– Preparation for interpretation and
generalization
• Quantitative analysis– Descriptive statistics– Inferential statistics
• Qualitative analysis
Empirical Methods – G. Grote ETHZ, Fall08
Basics on inferential statistics (1)
• Overall objective:– Estimating the likelihood of results found in
a particular sample occurring in the relevant population
– Accuracy of this estimation depends on sample size and variance of variables studied
– Error probability is controlled for both adopting a false hypothesis and rejecting a correct hypothesis
Empirical Methods – G. Grote ETHZ, Fall08
Basics on inferential statistics (2)
• Steps:– Determining null hypothesis and alternative
hypothesis – Calculate estimate for relevant parameter
(e.g. mean)– Calculate and compare test statistic and
critical value– Decide for/against null hypothesis
Empirical Methods – G. Grote ETHZ, Fall08
Basics on inferential statistics (3)
• Select appropriate tests– Based on measurement level of variables
(nominal, rank, interval, ratio)– Based on distribution of variable
(e.g. normal distribution) – Based on research question
• Comparing means (two groups, e.g. t-test; more groups, e.g. analysis of variance)
• Testing associations (e.g. correlation, regression)