Achieving Learning Objectives Online: Not All Platforms are Equal!

Nick FeamsterGeorgia Tech

A Tale of Two MOOCs

• A free Coursera MOOC on Software Defined Networking to over 50,000 enrolled students– Scalable delivery, assignments, and forums

• A for-credit Udacity “MOOC” on Computer Networking for Georgia Tech’s Online MS program, to 200+ enrolled students– Not so scalable…

Subtle differences in platform design have significant effects on a course’s ability to scale.

MOOC #1: Software Defined Networking

• Extremely hot, new topic– No existing “real world” courses on the topic

(yet)– A chance to develop an archetype course

(and material) that others might use in their own courses

– I wanted to be copied. This seemed like a good way.

• Boutique topic means that there were no “set in stone” ways of teaching the course– Easier to think outside the box, since there

was no existing material anywhere

http://blog.sflow.com/2012/05/software-defined-networking.html

Two Offerings So Far:• Summer 2013

(6 weeks)• Summer 2014

(8 weeks)

Who Takes The Course?

• 50,000+ people registered each year– 10,000 people active

each week– 1,000-3,000 watching

lectures(3,500+ in 2014)

• 500+ turn in programming assignments each year

Other Demographics (Summer 2014)

• 55,000+ Enrolled• 197 Countries– 31% from the United States– 15% from India

• Sex and Age– 86% Male– 74% 20-39 years old

• Education and Employment– 58% Employed full-time– 26% Full-time students– 38% have Undergrad degree, 31% have Masters

What the Instructor Sees

How Was the Course Developed?• Course lesson plan, with learning objectives• Top down design: – Figure out modules, continue subdividing until you

get 10-minute “lectures”• Production– Lecture Filming and Production– Assignments– Syllabus

• Operation– Forums

Filming Take 1: Studio

• Started in a studio at Georgia Tech• This quickly proved intractable– Studio is distracting. Lots of people watching

makes recording nerve-wracking.– Studio time is limited, contentious, and expensive.– Post-production quality was not to my liking.– Faculty members need scheduling autonomy.

• A lot different from a TV interview!

Filming Take 2: Camtasia

• Studio people will claim that quality suffers.• I believe this is bogus, for several reasons.– People can take their time recording, take breaks, record

when they are “in the zone”.– Screen captures from a laptop are easy.– Recording quality from a good laptop is quite fine.– Nobody has ever complained about the quality of the

recordings. Content is what matters most.

• Takes a little getting used to at first (looking at the camera, etc.)

• You have to do your own editing.

Producing Lectures in Camtasia

• General rule: 10-15 minutes of lecture == 2 hours of recording and editing– This does not include time to prep slides, etc.

• This is way different than a usual lecture!– Biggest lesson: Silence is quite easy to edit!– Stumbling is OK: Just say the same thing again– You can also “practice” while the video is rolling

• Need to find a quiet, well-lit location• Takes patience, but it starts to be quite fun

General Lecture Production Process• Determine learning objectives• Research the topic • Write slides– General format: Summary/outline, 7-10 content slides,

wrap-up

• Plan demonstrations– Plan on-screen landscape

• Record (one sitting for consistency…big challenge!)• Edit and export• Upload• Note: You can prepare lectures out-of-order!

New Modes of Lecture Production• On-screen demonstrations

– Videos lend themselves very nicely to on-screen demonstrations

– User can see things in “life size”– User can pause, rewind, etc. Very helpful

tutorials for working through assignments• Interviews with luminaries in the field

– Google “On Air” Hangouts– Can be done anywhere, and people are

willing!– Process: Invite, script questions, send,

revise. Stay roughly on script– Many people have since copied this idea

(Nick McKeown, Alex Orso, Mike Hicks, etc.)

Lessons from Lecture Production• Lots of money is not needed. Students want

content, clearly delivered. Camtasia works.• A 45–60-minute lecture is about 15 minutes of

well-polished video. Content distills well.• Lecturer is always “ahead” of the students.

This creates interesting time-shifting effects, with advantages and disadvantages.

• MOOCs are asynchronous and autonomous for the student. This is a big reason for their success. Production should be the same.

How Was the Course Developed?• Course lesson plan, with learning objectives• Top down design: – Figure out modules, continue subdividing until you

get 10-minute “lectures”• Production– Lecture Filming and Production– Assignments– Syllabus

• Operation– Forums

Developing Scalable Quizzes: Stick to the Basics

• Problem: The platform is not very good at handling anything that’s not multiple choice or simple numerical answer.– Short answers become massive regular expressions– …will likely improve as the software gets better

• Stick to multiple choice if possible• Main differences from a normal MC quiz– Helps to write up explanations for each option– Need to figure out if students can re-take the quiz, and if

so, how many times

Developing Assignments is Hard• Wanted to teach concepts and programming– Not a “heavy-duty” programming course, but

some concepts are best explained and learned through short code exercises.

• Code submission possible, but grading must be done on regular expression evaluation of output

• Problem: How to guarantee uniform programming environments?

Uniform Environment: Virtual Machine

• Asked students to:– Install VirtualBox (or equivalent) virtual machine emulator– Download 64-bit VM image that had everything pre-

packaged (no asking students to install software themselves). Kind of like a lab at home.

• This has generally worked. A few problems– VM is huge (~1 GB), so have to get it right the first time– Not everyone has a 64-bit machine– Certain performance, even when run in an emulator, varies

depending on underlying hardware

How Was the Course Developed?• Course lesson plan, with learning objectives• Top-down design: – Figure out modules, continue subdividing until you

get 10-minute “lectures”• Production– Lecture Filming and Production– Assignments– Syllabus

• Operation– Forums

Communicating with 10,000 Students Can Scale!

• I feared absolute disaster– Small glitches in

assignments in a classroom of 50 students create mayhem.

– I feared that I would be spending my life answering minutia in online forums.

• Thankfully, this did not happen!• Lesson: Giving an assignment

to 10,000 students in a MOOC is easier than giving it to 50 in a classroom!

Why Assignments Can Scale• First of all, you do your best to work

out the bugs, mostly out of fear – We attempted everything that

we assigned, sometimes on multiple platforms

• More importantly: Self-selection!– Everyone doing the assignments

wants to be there– This makes a huge difference– People not only fix issues

themselves, they rewrite documentation

• Caveat: Course difficulties vs. platform difficulties are currently hard to tease apart. Some of this is “user error”.

A student’s blog post describing part of the assignment setup in detail…

Some Feedback…

A Tale of Two MOOCs

• A free Coursera MOOC on Software Defined Networking to over 50,000 enrolled students

• A for-credit Udacity “MOOC” on Computer Networking for Georgia Tech’s Online MS program, to 200+ enrolled students

“MOOC” #2: Online MS at GT

• Online version of CS 6250: Computer Networking

• Two offerings thus far (Spring and Summer 2014)– About 120+ students in each offering– Paying tuition for a “degree”– Not specialists in networking

• Technical difference: Platform

How Was the Course Developed?• Top-down planning, with “scripts”• Several full-time staff members– Course developer from Udacity– Video editor from Udacity

• Part-time from others– Three paid TAs/Graders– Many other people handling logistics

• Course recorded in studio, ahead of time, with custom equipment– Recording done in “marathon stints”– Not possible to integrate interviews

• No auto-grading, little visibility of content, etc.• Initially, all material released at once (huge mistake)

Comparing the ApproachesCoursera (SDN MOOC)

Udacity (OMS “MOOC”)

Video Delivery Camtasia-recorded videos, talking head and slides + on-screen demos

Tablet writing + moving hand

Grading Simple MC quizzes, auto-graded programming assignments.

No auto-grading! TAs graded assignments “manually”.

Forums Initially “home brew”. Moved to Piazza.

Piazza, but the students expect more direct instructor engagement.

Pacing One week at a time, synchronous (though people fall behind)

All-at-once release.

Demographics Professionals and students taking the course for free, for enrichment.

Paying students who are not necessarily interested in the subject. Entitled.

Cost 300+ hours of my time, 40+ hours of TA time (all “Free”)

Estimated: $150k (staff salaries, studio time, etc.)

Leave the Pedagogy to the Instructors

• Substance trumps style.

• A MOOC platform should allow instructors to deliver the content in whatever format they see fit.

• Mandating a particular “style” emphasizes style over substance.

GT OMS Degree “MOOCs” Are Neither Massive nor Open

• One of the important features of MOOCs that allows scaling is community support.

• Things that dilute this support:– Students who pay and expect staff contact– Students who are not interested in the content,

but rather trying to get a degree– “Go at your own pace” style (students are on their

own, course staff can’t keep up)• Other platform-specific factors prevent scaling– Instructor has less autonomy in creation and

management of content– Support for auto-grading is critical!

Lesson #1: MOOCs Change the Game for Instruction

• Everyone is watching you– In seven years, never got feedback (or

acknowledgment) about on-campus course– Now, everyone’s eyes are on you– Huge pressure to do a good job

• MOOCs are a “forcing function” for introducing new media into the classroom– I wouldn’t have thought to do “on air” interviews for a

classroom; people would likely decline them anyhow– I will likely re-use some MOOC content in the “real”

classroom

Lesson #2: Time Investment Pays Off• Typical Course Creation (45-minute lecture)

– 1-2 hours to create slides– 1 hour to deliver the lecture– Number of students: 50-100– Total: 3 hours per lecture– … but then you have to do it again! (can’t save an awesome

lecture, students can’t rewatch, etc.)• MOOC Course Creation (10-minute video)

– 1-2 hours to create slides– 2-3 hours to produce the lecture– Total: 3-5 hours per lecture– Number of students: 500-100,000– … but, you have the archive. Can keep improving it, use it to

enhance in-classroom discussion, etc.

Lesson #3: Certain Aspects of MOOCs Scale Surprisingly Well

• Lectures are watchable by thousands– The scale can be a motivator for guest lectures!

• Programming assignments can be graded automatically, if designed well

• Self-motivated, self-selected students fix glitches, help each other out

Conclusions• Scalable MOOCs require– Community engagement– Low barriers to creating content– Support for a diversity of delivery modes to match

the material• Not all platforms are equal– A platform should give the instructor autonomy

and visibility.• For-credit “MOOCs” face far different scaling

challenges than true MOOCs.– At the moment: neither massive, nor open