Big Qualitative Data, Big Team, Little Time – A Path to PublicationFebruary 3, 2016

Dr. Charlotte Clark, Duke UniversityNoelle Wyman Roth, QSR International

Or, how to manage a large team project with a lot of data in a short period of time.

Overview & Project

Data Collection & Management

Coding Workshop

Findings

Lessons Learned

Overview & Timeline

January 14

Retained; Funding from Bill & Melinda

Gates Foundation

February – Early MarchMarch 10-14

March 15-16

Planning

Training & Intensive Coding

Workshop

Submission

March 17-31

Query & Analysis

March 31

Writing & Editing

Peer-to-Peer Writing in Introductory Level MOOCs

*Source: Wikipedia

• Online course aimed at unlimited participation and open access via the web* English Composition I: Achieving Expertise

Introduction to Chemistry

Massive Open Online Course (MOOC)

• evaluate how peer-to-peer interactions through writing impact student learning in introductory-level massive open online courses (MOOCs) across disciplines.

• compare humanities and natural sciences class.

Project Aim

• peer-to-peer interactions in writing through the forums and through peer assessment enhance learner understanding, link to course learning objectives, and generally contribute positively to the learning environment.

Results

Data Collection & SamplingTwo sources of data from both courses:1. Discussion Forum

• Weekly forum posts in each course

• Top posters in each course• General discussion forum

2. Peer Assessments

Data type Hours Basis estimate

PIT Threads 60Basis: 35 threads per PIT x 7 PIT = 240 total threads. Estimate 15 minutes on average per thread to code.

Top 3 posters in threads 24

Basis: 4 hours per student, 3 top posters in 2 disciplines. Code no more than 50 posts per student; not looking at threads they initiated, nor at threads to which they posted for context. Only coding their own words.

Peer Review Chemistry 8 Basis: 50 students, 10 minutes per studentPeer Review Eng IAAW 30

Basis: 2% of 8800 threads; 10 minutes to code each thread

Peer Review Eng Procrastinators 15 Basis: 125 threads; 7 minutes to code each thread

Peer Review Eng Student portfolios 25

Basis: 50 students (match to Chemistry), 30 minutes per student (2 minutes per eval of project draft and final x 8 + 15 minutes for student reflection)

Data Collection & Sampling Chemistry English

CompositionTotal

Sources Posts Sources Posts Sources PostsTop Posters 3 85 3 209 6 294Forums 124 1344 206 1051 330 2385

General Forums 25 809 37 86 63 895Points in Time 99 535 133 768 232 1303

Week 1 29 163 36 106 65 269Week 4 35 164 35 289 70 453Week 7 35 208 27 169 62 377Week 12 N/A N/A 35 204 35 204

Peer Review Writing assignment on forums

106 370 96 325 195 695

Student Portfolios N/A N/A 40 N/A 40 N/APeer Evaluations N/A N/A 279 N/A 279 N/ASelf-Evaluations N/A N/A 39 N/A 39 N/A

Total: 592 sources; 3374 posts

Challenges • Team of 9 coders, including PIs• Little to no experience with

qualitative data analysis• New to NVivo

• One week timeline for data collection, coding, & analysis

• Tons of data• Multiple data types • No data gathered

• Developed a plan • Coders gathered data based on

assignments• Trained only on what need to

know• Created a schedule • Tracked progress carefully• Used multiple tools to manage

data & collaboration

Solutions

Why NVivo Worked Well• Go through an intercoder

reliability process. • Talk to your teammates

frequently.• Keep good notes and track

decisions.• Develop a team protocol. • Maintain individual research

journals.• Keep a codebook. • Be iterative and flexible.

Merge projects Coding comparison query

Coding stripes by user

Memos See also links

Node description boxModify list view

In NVivo:In General:

Create NVivo File

Merge projects; compare coding

Distribute to coding team

Master File

Coder 1

Master File

Coder 2

Continue Analysis Coder 1 Coder 2

Collaboration in NVivo

Coding Workshop Timeline• Developed coding protocol• Created master NVivo file

Prior to Workshop

• Morning: introduced project, manual coding, collaborative coding

• Afternoon: ~3 hour NVivo trainingDay 1

• Morning: team; import & code data• Afternoon: Intercoder reliability

discussion; independent coding Day 2

Coding Workshop Timeline• Individual coding assignments• Daily team discussion/check-in• Set goals & tracked progress

Days 3-4

• Analysis discussion • Cleaning data for coding

inconsistenciesDay 5

• Cross-tabulate & query based on research questionsDays 6-7

Finding: Student Discussions Reflect Course ContentCourse Week 1 Week 4 Week 7 Week 12

Chemistry

English Composition

Finding: General & Discrete Learning Gains in Forums

Learning Gains Demonstrates what has learned

Evidence of incorporat -ing feedback

Improved Grades

Understanding

Learning Gains in Chemistry ForumsLearning Gains

Demonstrates what has learned

Evidence of incorporat -ing feedback

Improved Grades

Understand-ing

Learning Gains in English Composition Forums

“I was stuck with the idea that my introductions should be one paragraph long.

Maybe I should experiment with longer introductions.”

“And I feel comfortable enough with the chemistry, the basic chemistry, to not avert my eyes like I used to. Whenever I saw a chemical equation I just, oh

well, never mind, and I’d just skip it.”

“I don’t know about you, but I’ve already learned an amazing amount from this class!”

Finding: Learning Gains in Peer Review

Demonstrates what has learned

Evidence of incor-porating feedback

Understanding

Learning Gains in Chemistry Peer ReviewLearning Gains

Demonstrates what has learned

Evidence of incorporating feedback

Learned through providing feedback

Understanding

Learning Gains in English Composition Peer Review “I found peer comments and their assessment invaluable.”

“I am, however, grateful for the kind parts of your review, and willingly admit to faults within the essay, although until this week, I was, like my fellows, unaware of the expected work on electron transits. By the time I did become aware of this, it was too late to make alterations! Thank you for a

thoughtful review.”

“Even more important bit I learned was the importance of feedback.

Feedback provides an opportunity to rethink the project, and dramatically

improve it.”

Finding: Attitudinal Summary Positive

Negative

Neutral

Attitude in Chemistry

Positive

NegativeNeutral

Attitude in English Composition

“I am starting to understand why I am studying on a Friday evening for the first

time in my entire life. :)”

Positive Attitude

“Go for it (un-enrole) [sic]- [two names removed]. You both know too much already and you obviously have nothing to gain from

this course. You’ll be doing us “stupid” students a favor.”

Negative Attitude

Lessons Learned & Limitations

Identify what is need to know

Keep the lines of communication

open

Rigorous & deliberate

planning is key

Limitations to NVivo merging

process

qsrinternational.com