qualmethods minicourse
TRANSCRIPT
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 1/50
Using Qualitative MethodsUsing Qualitative Methodsin Empirical Studies of in Empirical Studies of Software EngineeringSoftware Engineering
University of Maryland Baltimore County University of Maryland Baltimore County Fraunhofer USA Center for Empirical Software Fraunhofer USA Center for Empirical Software
Engineering Maryland Engineering Maryland
ESELAW 2009 ESELAW 2009
12 November 2009 12 November 2009
São Carlos São Carlos- -SP, Brazil SP, Brazil
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 2/50
DefinitionsDefinitions
Qualitative data - data in the form oftext and pictures, not numbersQualitative analysis – analysis of
© Carolyn Seaman, 2009
trends, patterns, and generalizationsGrounded theory – theory formedbottom-up from the (usuallyqualitative) dataRich data – data that includes a lot of
explanatory and context information
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 3/50
Why Qualitative Methods?Why Qualitative Methods?
Problem : Difficult to answer complexSE questions with a purelyquantitative approach because
© Carolyn Seaman, 2009
Typically have small sample sizesExperiments are expensive to runNeed some support for a hypothesis
before investing effort in full experimentSolution : Use a qualitative approachthat includes a quantitative aspect
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 4/50
Types of resultsTypes of results
A qualitative study will result in:Propositions tied to a trail of“evidence”
© Carolyn Seaman, 2009
-Complex findings that incorporatethe messiness of the phenomenon
under studyExplanationsAreas for future study
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 5/50
Types of ResearchTypes of ResearchQuestionsQuestions
Qualitative methods are most appropriatewhen:Subject of study involves human
© Carolyn Seaman, 2009
e av orNo concrete hypothesesVariables hard to define or quantify
Little previous workQuantitative results may be hard tointerpret
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 6/50
Advantages to ResearchersAdvantages to Researchers
Richer resultsResults more explanatory
© Carolyn Seaman, 2009
Avoid errors in interpretation
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 7/50
Advantages to PractitionersAdvantages to Practitioners
Richer , more relevant resultsTerminology of results
© Carolyn Seaman, 2009
Opportunity to clarify and explainfindings
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 8/50
Overview of TechniquesOverview of Techniques
Data Collection
Prior Ethnography
Data Analysis
Codin
© Carolyn Seaman, 2009
ParticipantObservationInterviewing
SurveysDocument Analysis
ConstantComparison MethodCross-case analysis
Member checkingAuditing
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 9/50
Participant ObservationParticipant Observation
Definition: non-covert directobservation of phenomenon
© Carolyn Seaman, 2009
inspection meetingscollected both qualitative and quantitative
datadid not participate in the inspectionused data forms as well as field notes
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 10/50
Observation Data FormObservation Data FormInspection Data Form
Class(es) inspected Inspection date: Time:
Author:Moderator:Reviewers:
Name Responsibility Preparation time Present
© Carolyn Seaman, 2009
Amount of code inspected:Complexity of classes:Discussion codes:D = Defects Q = Questions C = Classgen defect U = Unresolved issues G/D = Globaldefects G/Q = Global questions P = Process issues A = Administrative issues
M = Miscellaneous discussion
Time logged (in minutes):D______ Q_____ C_____ U_____ G/D______ G/Q______ P______ A______ M______
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 11/50
Field Notes ExampleField Notes Example
The "step" function is a very important but complicated function.[Reviewer1] did not have time to review it in detail, but[Author] said he really wanted someone to go over it carefully,so [Reviewer1] said she would later.
-
© Carolyn Seaman, 2009
values. This is a problem because often the code is such thatthere is no way to tell what a particular variable was initializedto. [Reviewer2] said "I have no way to see initial value". Thiswas a global discussion, relevant to many classes, including
[Reviewer2]’s evidently.
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 12/50
InterviewingInterviewing
Interviews are good for gettingopinionsfeelin s
© Carolyn Seaman, 2009 12
goalsprocedures (both formal and informal)
not facts
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 13/50
Standard Interview FormatsStandard Interview Formats
Structured (standardized)Tightly scripted, almost verbalquestionnaire
© Carolyn Seaman, 2009 13
,Analyze like questionnaire“How many times a day do you access the internet? [0, 1-5, 5-10, 10-15, 15+]”
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 14/50
Standard Interview FormatsStandard Interview Formats
Unstructured(Open/Informal/Conversational)Guided by a very scant script.Rich but not re licable.
© Carolyn Seaman, 2009 14
Difficult to be systematic, problem ofcoverage.Minimize interviewer effects, preservesinterviewee point of view.Interviewee led, interviewer probes.“Please, tell me about your internet usage...”
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 15/50
Standard Interview FormatsStandard Interview Formats
Semi-structuredGuided by a script ( interview guide ), butinteresting issues can be explored in
© Carolyn Seaman, 2009 15
.Good balance between richness andreplicability.Mixed analysis techniques.
“In a typical day, how often do you use the internet?”
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 16/50
Interview questionsInterview questionsClosed
Predetermined answer format (e.g. Yes/No)Easier to analyze
OpenNo predetermined answer format
© Carolyn Seaman, 2009 16
More complete responseCombinationClosed, with opportunity to elaborate
Probes
Pitfalls :leading questionsdouble-barreled questions
judgmental questions
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 17/50
Interview GuideInterview Guide
A script for use by interviewer only“Wish list” vs. structuredFlow/direction to interview
© Carolyn Seaman, 2009 17
equ re top csTransitions between topic areasImportant for replicabilityWording and sequence are critical
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 18/50
Interview DesignInterview DesignConsiderationsConsiderations
Context switchingFlow between open and closed
© Carolyn Seaman, 2009 18
“Shape ” of interviewMost important stuff first
Wording
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 19/50
Interview ShapesInterview Shapes
FunnelBegin with open, gradually become more closedGood if you’re not sure what you’re going to get
Pyramid
© Carolyn Seaman, 2009 19
,Good with nervous interviewees
Hour glassBegin with open, gradually become more closed,then open up again at end to pick up things youmight have missedGood if you know what you want, but suspectthere are important things you don’t know aboutyet
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 20/50
Interview Guide 2a : In-depth project interviewsWho : Developers on [Project1], [Project2], [Project3]Subjects covered : general opinions of GSS processes and productsDuration : 60-90 minutes
What do you like about the current process using GSS?
Example Interview GuideExample Interview Guide
© Carolyn Seaman, 2009
at o you s e a out t e current process us ng
Do you depend on any other groups, either for information or help withGSS, or for work to be done related to GSS?
What do you like about the applications resulting from using GSS?
What do you dislike about the applications resulting from using GSS?
Have there been any problems with the interface between GSS and otherCOTS products?
What do you see as the top risks associated with the use of GSS? Howwould you mitigate these risks?
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 21/50
Interviewing PointersInterviewing Pointersgive clues about the level of detail you want
establish rapport , but be subject neutralavoid jargon , esp. academesedispel any notion of the “ right ” answer
la the novice when a ro riate
© Carolyn Seaman, 2009 21
probe , but do not leadalways be aware of your biasesbe sensitive to their work (environment/schedule)no more than 60 minuteslet interviewee know next stepsend with “ anything else I should know? ”say Thank you !
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 22/50
Recording of interviewsRecording of interviews
Audiorecording
© Carolyn Seaman, 2009 22
Scribing
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 23/50
AudiorecordingAudiorecording
Best memory mechanismFull transcription or just verbatim quotesStill take notes
Tapes fail, digital files are deleted
© Carolyn Seaman, 2009 23
Does not record all aspects (esp. context / facial expressions)Required consent
Always ask first.
Do NOT hide recorder, keep it visible at alltimes.Give the option to turn it off at any point.
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 24/50
NotetakingNotetaking
Very hard to take notes andinterview at the same timeThere are some super-
© Carolyn Seaman, 2009 24
researchers who can do itInevitably results in incompletenotesSlows down the interviewSometimes inevitable
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 25/50
ScribingScribing
Partner-based interviewingAdvantages of a single contact vs. trading-offCan share roles (interviewer/scribe)
© Carolyn Seaman, 2009 25
ta e notes, t oug to erentdegree
Group debrief: what did you get/miss?Synchronize notes: overlap and emphasis
Clarify while it is still in your head
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 26/50
Writing up the interviewWriting up the interview
© Carolyn Seaman, 2009 26
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 27/50
Interview NotesInterview Notes
Write it up immediatelyDescriptive vs. reflective notesUse Observer’s Comments
Impressions, state of mind, assumptions, notes to
© Carolyn Seaman, 2009 27
How detailed?Verbatim transcript
only possible with audiorecordingExtremely labor-intensive
Summaries with major points quotedOK, but use LOTS of quotesStart closer to verbatim at the beginning of a study
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 28/50
Interviewing ExerciseInterviewing Exercise
Background :The National Federation of Makers of Feijoada(FNFF ) is concerned that the nationalconsumption of feijoada is declining due to
© Carolyn Seaman, 2009
ecreas ng qua ty o e oa a .So they have asked us to interview the topfeijoada chefs in the country (as determined byregional competitions)The goal is to find out the secrets to masterfeijoada making, so that it can start to be taught inelementary schools .
28
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 29/50
Interviewing ExerciseInterviewing Exercise
Three versions of the interviewguide
© Carolyn Seaman, 2009
You will be the interviewees
So take a moment to think of your
favorite feijoada chef
29
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 30/50
Interviewing ExerciseInterviewing Exercise
RecapFirst interview: pyramid
Started with easy, closed questionsEnded with open-ended questions
1. What do you think makes your feijoada the best?2. What is special about your ingredients?3. What are the basic steps to making feijoada?4. Who taught you to make feijoada?5. How long does it take you to make a feijoada?
1. How often do you make feijoada and how long does ittake you?
2. What do you think makes your feijoada the best?3. Of course, you always wash your hands thoroughly
before you start, right?
© Carolyn Seaman, 2009
econ nterv ew: funnelStarted very broadly, with open questionsFollowed up with narrower, closed questions
Third interview: just bad
Leading, judgmental questionsDouble-barreled questionsSwitching from topic to topicSwitching between open and closed
30
1. What is your name, please?2. How often do you make feijoada?3. How long does it take to make feijoada?
4. What are the ingredients you use?5. What do you think makes your feijoada the best?
. the end of the cooking?
5. What kind of pot do you use?
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 31/50
Constant ComparisonConstant ComparisonMethodMethod
Qualitative analysis method
Meant to generate grounded theoryOperates on a set of field notesB sic rocess:
© Carolyn Seaman, 2009
codinggroupingwriting field memo
forming hypothesesRepeated periodically in parallel withdata collection
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 32/50
What’s a Code?What’s a Code?
A labelA concept
© Carolyn Seaman, 2009 32
A categoryA relationship
A theme
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 33/50
What’sWhat’s Coding?Coding?
Open coding - assigning codes topieces of textual dataCoded “chunks” can overlap
© Carolyn Seaman, 200933
One chunk can have several codesAxial coding - grouping, categorizing,combining coded chunks
Selective coding - making sense of it
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 34/50
What’s here? What are the pieces?
Identification/discovery of conceptsClassification labelin of henomena
Open CodingOpen Coding
© Carolyn Seaman, 200934
Abstraction (this is part of that)Comparative analysis (this is different fromthat)
Categorization (organization, grouping)Value-neutral, at least initially“complexity” not “high complexity” or “lowcomplexity”
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 35/50
Preparing for codingRead the dataRead background material and research designCreate pre-formed codes, if applicable
Coding by hand
Open CodingOpen Coding ProcessProcess
© Carolyn Seaman, 200935
, .
Photocopy, scissors, and envelopesMS Word commentsExcel
Coding tools – NVivo, Atlas TI
Coding schemePre formed or post formed codesConstant iterationStructure develops over time
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 36/50
• Background :
• Study of the role of documentation insoftware maintenance
• Interviews with experienced software
OpenOpen Coding ExerciseCoding Exercise
© Carolyn Seaman, 200936
ma nta ners n severa organ zat ons• Process :
• I’ll show you an example• Then you’ll try it – code one excerpt with
one code• Find a partner – compare your codings• I’ll show you my coding of the excerpt
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 37/50
Respondent BackgroundInformation Gathering
Transition to maintenanceTypes of documentationCharacteristics of DocumentationQuality of documentation
Coding SchemeCoding Scheme
© Carolyn Seaman, 200937
Properties of documentationMissing documentationCreating documentationLocation of documentationImportance of documentationHuman sources of informationQuality of ProcessGreat Quotes
Human Sources of Information
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 38/50
Open Coding andOpen Coding andQuantificationQuantification
One form of coding
Objective is to derive quantitative data from qualitative datafor future statistical analysisUsually involves counting
How many subjects said…?
© Carolyn Seaman, 200938
…How many times did …?
Or timingHow long did subjects spend doing…?How long did it take to …?
Inevitably loses richnessOften seems a little like missing the point
What’s the point of collecting rich data when you’re just goingto condense it down to numbers?
But often is an effective and necessary way to reduce thesize of the data
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 39/50
© Carolyn Seaman, 200939
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 40/50
© Carolyn Seaman, 200940
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 41/50
How are things related?
Initial process of reassemblingRelationships among categories and codes
Axial CodingAxial Coding
© Carolyn Seaman, 2009 41
Structure (why?)Process (how?)Explanations not causal prediction
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 42/50
How does it all fit together?
Also called sense makingRelationships among relationships
Selective CodingSelective Coding
© Carolyn Seaman, 2009 42
Theory constructionThe central categoryStoryline memosRole of literatureWrite, write, write!!!Field Memos
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 43/50
Field MemosField Memos
The “ single most powerful analytical tool ” forqualitative researchersSimply, a piece of writingMaybe will later become part of a report,
© Carolyn Seaman, 2009 43
Summarizes and synthesizes:
A propositionAn open questionA chain of evidence and logic
The complexity of a conceptRich descriptionVersion control and organization
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 44/50
Judging ValidityJudging Validity
Validity of methodsTriangulation
© Carolyn Seaman, 2009
Contradictory evidenceWeight of evidence
How much is enough?Variety as well as quantity ofevidence
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 45/50
Using Qualitative andUsing Qualitative andQuantitative MethodsQuantitative Methods
Together Together
Qualitative and quantitative methodsbest used in combination
© Carolyn Seaman, 2009
an s mp y e use n para e oaddress the same researchquestions
There are other strategies to betterexploit the strengths andweaknesses of the methods
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 46/50
Example Design 1Example Design 1 :: StatisticalStatisticalHypothesis Testing withHypothesis Testing withFollowFollow- -up Interviewsup Interviews
Classic design – often done withoutfully exploiting the interview data
© Carolyn Seaman, 2009
Example scenario:Blocked subject-project experiment toevaluate a new testing techniqueStatistical results show that technique ismore effective on some applications thanon othersQualitative results show why
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 47/50
Want to evaluate a new technique, butnot sure what the evaluation criteriashould be
Example Design 2 : Using GroundedTheory to Identify Variables
© Carolyn Seaman, 2009
Evaluating a collaborative design processUse participant observation of designmeetings to generate hypotheses aboutproperties of the resulting designsGrounded hypotheses are used to design aquantitative evaluation of the resultingdesigns
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 48/50
Example Design 3Example Design 3 :: Using Prior Using Prior Investigation toInvestigation to
OperationalizeOperationalize VariablesVariables
Relevant variables are known, but therange and types of values is difficult tospecify
© Carolyn Seaman, 2009
Example scenario:Want to study the relationship betweendeveloper experience and types of defectsFirst use interviews to identify the range of
developer experience (in its complexity) anda taxonomy of defect typesQuantitative study then is much moreeffective when using this operationalization
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 49/50
ConclusionsConclusionsEmpirical software engineering researchers areaddressing more and more complex research
questions that have increasingly humanelementsQualitative methods, usually in combination with
© Carolyn Seaman, 2009
,
this complexityQualitative methods are both flexible andrigorousQualitative analysis provides richer, morerelevant, and more explanatory resultsThe most effective research designs combinequalitative and quantitative methods
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 50/50
BibliographyBibliographyDay, I. Qualitative data analysis: A user-friendly guide for social scientists . New York: Routledge. 1993.Eisenhardt, Kathleen M. "Building theories from case studyresearch." Academy of Management Review, 14:4, 1989.Gilgun, J.F., "Definitions, Methodologies, and Methods inQualitative Family Research." Qualitative Methods in Family
Research . Sage, 1992. pp. 22-29Glaser, Barney G. and Anselm L.Strauss. The Discovery of Grounded Theory: Strategies for Qualitative Research. de
© Carolyn Seaman, 2009
, .
Jick, Todd D. "Mixing qualitative and quantitative methods:Triangulation in action." Administrative Science Quarterly,24:4, 1979.Judd, Charles M., Eliot R. Smith, and Louise H. Kidder.Research Methods in Social Relations. Harcourt, Brace,Jovanovich, 1991.Miles, Matthew B. and A. Michael Huberman. Qualitative DataAnalysis: An Expanded Sourcebook. Sage Publications, 1994.