Using intelligent tutorsUsing intelligent tutorsto enhance student learningto enhance student learningof application programming of application programming

interfacesinterfaces

Aniket Dahotre, Vasanth Krishnamoorthy, Matt Corley, Chris ScaffidiSchool of Electrical Engineering and Computer Science

Oregon State University

22

Application Programming Interface Application Programming Interface (API)(API)

“The most serious problem facing instructors who try to teach Java—or any modern industrial-strength language for that matter—is the problem of scale. While such languages may themselves be reasonable simple, writing any useful programs requires the use of classes supplied as application programmer interfaces (APIs) along with the language. For modern languages, such

API collections are vast. The existence of these huge libraries makes it difficult for students and teachers to learn the language without suffering from conceptual overload.”

--ACM Java Task Force, 2004

Introduction Approach Evaluation Conclusion

33

ExamplesExamples(where API here = task-focused collection of methods)(where API here = task-focused collection of methods)

APIs for common tasks APIs for less common tasks

Reading from a file Setting permissions on a file

Writing to a file Encrypting a file

Downloading from a web server Embedding a browser in an app

Querying a database Storing a CLOB in a database

Launching a thread Reading stdout of a process

Parsing XML Transforming XML with XSLT

Displaying an image Reading pixels from a GIF image

Introduction Approach Evaluation Conclusion

44

Particularly challenging in Particularly challenging in project-based learning curriculaproject-based learning curricula

N students

N / x different projects

y * N / x different APIs

1 professor + 2 teaching assistants

y * N / x >> 3

Students end up struggling with many APIs on their own.

Introduction Approach Evaluation Conclusion

http://www.flickr.com/photos/kchichester/3388004334/

55

Related WorkRelated Work

• Empirical studies– Novice programmers search for code examples when they

struggle with using an API

– One survey: 40% of programmers complained about lack of examples; 28% about understanding documentation [Robillard 2010]

• Novel tools– Can search for code examples, copy them into the

programmer’s program, and help to customize them

– Programmers sometimes don’t understand code they copy

Introduction Approach Evaluation Conclusion

66

ApproachApproach

• Let the professors continue to focus on teaching students to use common APIs• Most commonly-used APIs

• APIs most closely related to the course’s core concepts

• Provide intelligent tutoring systems to help students learn the rest of the “specialized” APIs• Delegate creation of intelligent API tutors to TAs

• Provide a tool so TAs they use existing code examples

• Store intelligent API tutors on website, so they can accumulate over multiple terms

Introduction Approach Evaluation Conclusion

77

Current paperCurrent paper

• How to create intelligent API tutors?

• Can intelligent API tutors match textbook instruction?

Introduction Approach Evaluation Conclusion

88

How to create intelligent API tutors?How to create intelligent API tutors?

1. TA enters topic keywords

Search engines

Teaching assistant

Teacher tool

2. ApiTutor uses keywords to

retrieve web pages about topic

3. ApiTutor identifies, parses, compares, and filters code

examples

4. ApiTutor presents examples to TA for

further filtering, organizing, and

annotating

Repository5. ApiTutor publishes

intelligent tutor to repository website

Introduction Approach Evaluation Conclusion

99

1. TA enters topic keywords1. TA enters topic keywords

Introduction Approach Evaluation Conclusion

1010

2. ApiTutor uses keywords to retrieve 2. ApiTutor uses keywords to retrieve web pages about topicweb pages about topic

Teacher tool

Bing

Yahoo

Website

…<code> try {

Class.forName(”sun.jdbc.odbc.JdbcOdbcDriver”); //Or any other driver}catch(Exception x){

System.out.println( “Unable to load the driver class!” );}</code>…

Introduction Approach Evaluation Conclusion

1111

3. ApiTutor identifies, parses, 3. ApiTutor identifies, parses, compares, and filters code examplescompares, and filters code examples• Extract candidate code snippet from <code> or <pre> tag• Parse snippet into abstract syntax tree• Represent each snippet as TFIDF vector

– Vector entries correspond to names of packages imported, names of classes referenced, names of methods invoked, and list of classes for variables instantiated

• Compute similarity matrix (all pairwise dot products)• Select largest elements of matrix

Introduction Approach Evaluation Conclusion

1212 Introduction Approach Evaluation Conclusion

4. ApiTutor presents examples to TA 4. ApiTutor presents examples to TA for filtering, organizing, and annotatingfor filtering, organizing, and annotating

1313 Introduction Approach Evaluation Conclusion

5. ApiTutor publishes intelligent tutor to 5. ApiTutor publishes intelligent tutor to repository websiterepository website

Student chooses a

topic from th

e list

Student interacts with the adaptive quizzes and

examples

Method term selected by teacher

Method term selected by teacher

Hint messages stored by teacher

1414

Can intelligent API tutors match Can intelligent API tutors match textbook instruction?textbook instruction?

• Laboratory study• 10 computer science students

– Undergraduate or master’s

• Two randomly-selected tasks per participant– Five possibilities: StringBuffer, FileWriter, HashMap, XML

DocumentBuilder, and JDBC Connection classes

• Structure of each task– Part 1: learn about API from intelligent tutor or textbook

(randomly selected)

– Part 2: three-question fill-in-the-blank quiz

Introduction Approach Evaluation Conclusion

1515

ResultsResults

Introduction Approach Evaluation Conclusion

1616

ResultsResults

• Average scores– 45% with textbook

– 93% with intelligent API tutors

• 2-Factor ANOVA– Textbook vs intelligent API tutor: significant at P<0.01

– Among the 5 API tasks: significant at P<0.01

– No interaction at P<0.10 (intelligent API tutor roughly equally effective across tasks)

Introduction Approach Evaluation Conclusion

1717

ResultsResults

• Average scores

– 45% with textbook

– 93% with intelligent API tutors

– Significant at P<0.01

• Time that students chose to study before quiz– 15 minutes with textbook

– 10 minutes with intelligent API tutors

Better scores in less time

Introduction Approach Evaluation Conclusion

1818

Conclusions… and more questionsConclusions… and more questions

• How to create intelligent API tutors?– Mining examples from the web seems to work well

• Can intelligent API tutors match textbook instruction?– Yes—in fact, intelligent API tutors outperformed textbook

… sparking more questions…• Will TAs be able to create intelligent API tutors this way?

• And will their intelligent API tutors perform so well?

• And will the approach work for a broad range of APIs?

• And will it actually lead to learning if used in practice?

Introduction Approach Evaluation Conclusion

1919

What next?What next?

• Generalize to support other APIs and languages

• Test, test, test– Have TAs create the intelligent tutors

– For dozens of APIs

• Deploy and evaluate– In the classroom

– In distance learning

– In on-the-job training

Introduction Approach Evaluation Conclusion

2020

Questions?Questions?

http://www.flickr.com/photos/robertcupisz/2540742831/

Introduction Approach Evaluation Conclusion

Ultimate goal: Enabling students

to learn specialized APIs on demand

2121

Protocol detailsProtocol details

• 15 minutes to read the API’s training materials– Allowed to move on to the quiz earlier if desired

• Told students to focus on class and method names

• Provided at the start of the experiment with an example of a quiz so they would be prepared for fill-in-the-blank

• No human tutoring at all– Computer vs textbook

Introduction Approach Evaluation Conclusion