innovations in elearning 2 june 2010elearningsymposium.gmu.edu/2010/presentations 2010... ·...

$: Innovations in eLearning 2 June 2010elearningsymposium.gmu.edu/2010/Presentations 2010... · Innovations in eLearning 2 June 2010. ... of how the Def\nse Acquisition University \⠀䐀䄀唀尩$
Keeping Students Awake – Creating Active Learning Environments with Interactive

Classroom Technology

Innovations in eLearning2 June 2010

Learn. Perform. Succeed.

Outline

• Classroom questioning strategies

• Using technology with classroom questions

• DAU experience with integrating technology with classroom questions

2


Classroom questioning strategies

• Examples• Pre-tests to activate prior knowledge and…

– Get the students ready to learn– Can be tied to lesson objectives so instructor can

allocate time to various parts of lesson • Directed “knowledge” questions to check

comprehension of material– At end of lesson topics (every 10-15 minutes)

• “What do you think” questions to generate discussion and give learners time to share experiences

3

Presenter

Presentation Notes

Comments by Bill Fast: The pretest strategy is discussed by Ruth Colvin Clark in her book: Building Expertise: Cognitive Methods for Training and Performance Improvements, 3rd Edition, pages 144-145. As you’ll hear each of us say at least once throughout this presentation, the clickers are a tool. They do not deliver content and they cannot suddenly turn bad content into great content. So like with any good lesson, you must first start out with solid content and good questions. There are many theories about questioning techniques and how to use them. On your handout are some resources on questioning strategies. 3 example questions for this presentation, which are inserted in this presentation as they would be in a regular classroom lesson. How do questions change by using technology? As stated before, good questions are still good questions.


Using technology with classroom questions

4

• Good questions are still good questions• Helps structure class time by breaking up

material and adding interactivity• Especially helpful for new instructors

• Anonymity of SRS • All students participate – not just the most

out-going ones

Presenter

Presentation Notes

Questions aren’t magically great questions just because the instructor used a clicker. Students tend to stop paying attention after 10-20 minutes. Lose focus. Clickers add interactivity like with distance learning courses. Some students will never raise their hand to answer a question or will freeze when put on the spot. Clickers can be anonymous. No fear of looking dump in front of the whole class.


Example Question – Attitude

5

Presenter

Presentation Notes

Next – moving on to the DAU experience


DoD Experience … so far

• Research about other DoD trials

• Results of DAU student and instructor surveys (engagement, satisfaction, assessment)

6


DoD Classroom Performance Systems

24 Military Bases17 Federal Agencies23 Military Academies87 Nuclear Submarines3K Army, Navy, AF and

Marine JROTCprograms

. .

.

.

.

.

.

.. ....

. . . . .. ....

. .

.

.

.

. .

.

.

.. ....

. .

.

.

.

. .

.

.

.

. .

.

.

.

. .

.

.

.

. .

.

.

.. ....

. .

.

.

.

. .

.

.

.. ....

. .

.

.

.

. .

.

.

.

. .

.

.

. .

.

.

.

. .

.

.

.. ....

. .

..

.

. .

...

. .

...

. .

...

. .

...

. .

... . .

...

. .

...

. .

...

. .

...

. .

...

. .

...

. .

...

. .

...

. .… .

.. .

. .… .

.. .

. .… .

.. .

. .… .

.. .

. .… .

.. .. .… .

.. .

Over 12,000 instructors and

1 million students use eInstruction

training tools

Presenter

Presentation Notes

eInstruction was formed in 1988 Our headquarters are in Denton Texas, and we have 3 other offices – Cincinnati, OH, Columbia, MD and Scottsdale, AZ Our team consists of 285 staff members, and 350 certified representatives in all 50 states. We also have an international division that is represented in over 20 countries worldwide.


Trident Submarine Training Facility

6 classrooms, n = ~1,1008


Trident Submarine Training Facility

6 classrooms, n = ~1,1009


Navy Junior ROTC

13 NJROTC units, ~30 students per class

10


Provide a global learning environment to support a mission-ready Defense Acquisition Workforce that develops, delivers, and sustains effective and affordable warfighting capabilities.

DAU Mission

Impact acquisition excellence through:Acquisition certification and

leadership trainingMission assistance to

acquisition organizationsOnline knowledge sharing

resourcesContinuous learning assetsStrategic workforce planning


DAU Performance Learning Model

Knowledge SharingDAP - Online portal to Big A & HCI knowledgeACC - DoD's online collaborative communitiesVirtual Library - Online connection to DAU research collection

Training CoursesClassroom & online DAWIA, Core Plus, & Executive

Mission AssistanceConsulting - Helping organizations solve complex acquisition problemsTargeted Training - Tailored organizational trainingRapid Deployment Training - On-site & online training on the latest AT&L policies

Continuous LearningCL Modules - Online, self-paced learning modulesConferences - PEO / SYSCOM, Business Managers, DAU Acquisition Community Symposium

Formal & informal learning at the point

of need

Presenter

Presentation Notes

The Acquisition, Technology, and Logistics (AT&L) Performance Learning Model (PLM) is a visual representation of how the Defense Acquisition University (DAU) fulfills its mission in providing practitioner training, career management, and services to enable the acquisition workforce to make smart business decisions and deliver timely and affordable capabilities to the warfighter. Three Pillars of Workforce Development The PLM’s three pillars identify the primary types of acquisition training and development an acquisition professional receives over the course of his or her career—Defense Acquisition Workforce Improvement Act (DAWIA) core certification, Core Plus development, and executive and leadership support. The pillars represent how acquisition professionals progress through their career, from obtaining certification at level I, II, or III in acquisition career fields; to gaining specialized experience in a career field; to obtaining executive-level capabilities to run acquisition programs. Foundational Structure of Workforce Development Laying the foundation for the three pillars of workforce development are DAU’s services: training courses knowledge sharing continuous learning performance support. As the Defense Acquisition Workforce’s premier learning and development center, DAU aligns its training with the specific career field requirements as outlined by DAWIA. In addition, the university has taken innovative measures to ensure that learning and acquisition support are available beyond DAWIA certification, creating a total learning environment for career-long solutions at the point of need. The PLM depicts this versatile and encompassing learning environment, and what follows is an overview of DAU’s numerous services.


Defense Acquisition Workforce

As of 30 Sep 09

Career Fields ARMY NAVY/USMC AIR FORCE 4th Estate TOTAL Auditing 0 0 0 3,777 3,777Business, Cost Est., & Fin. Mgt 2,771 2,286 1,845 360 7,262

Contracting 8,391 5,516 7,443 6,305 27,655Information Technology 1,843 1,240 966 309 4,358Life Cycle Logistics 7,952 4,784 1,989 127 14,852Production, Quality & Manufacturing 1,930 2,064 389 4,640 9,023

Program Management 3,452 4,598 4,461 911 13,422SPRDE 10,412 18,328 7,248 1,339 37,327Test and Evaluation 2,235 2,833 3,630 194 7,892Other 1,370 5,323 203 639 7,535Total 40,356 46,972 27,174 18,601 133,103


Global Reach to Support Combatant Commands FY09

NORTHCOM ForeignStudents

U.S.Students

CL Grads 1,664 468,868Classroom Grads 20 38,631DL Grads 194 150,853

SOUTHCOM ForeignStudents

U.S.Students

CL Grads 50 533Classroom Grads 0 0DL Grads 15 33

CENTCOM ForeignStudents

U.S.Students

CL Grads 195 1,592Classroom Grads 7 22DL Grads 58 435

PACOM ForeignStudents

U.S.Students

CL Grads 1,212 11,432Classroom Grads 59 441DL Grads 233 1,616

EUCOM ForeignStudents

U.S.Students

CL Grads 884 5,342Classroom Grads 14 92DL Grads 67 460

AFRICOM ForeignStudents

U.S.Students

CL Grads 6 27Classroom Grads 0 0DL Grads 64 20

http://upload.wikimedia.org/wikipedia/commons/6/65/USCENTCOM.jpg�


DAU Experience with “Clickers”

• Mission Assistance: 6 events @ 500 students– Army Congressional Fellowship Program– Lean Six-Sigma Green Belt Training– ASMC Professional Development Institute

Faculty gained experience with 23 classes/events and ~ 1,000 learners

• Courses # Classes # StudentsACQ 201 2 72BCF 301 7 170LOG 350 4 100PMT 352 4 120

http://images.google.com/imgres?imgurl=http://www.niu.edu/blackboard/images/cpsrfHighEd2.jpg&imgrefurl=http://www.niu.edu/blackboard/resources/cps.shtml&h=319&w=167&sz=24&hl=en&start=29&tbnid=7Uen2ej5EBWzcM:&tbnh=118&tbnw=62&prev=/images?q=response+unit+einstruction+cps&start=20&gbv=2&ndsp=20&hl=en&safe=active&sa=N�

http://images.google.com/imgres?imgurl=http://www.turningtechnologies.com/images/lg_response_cards_RF.jpg&imgrefurl=http://my.simmons.edu/technology/media/equipment/ARS_at_Simmons.shtml&h=775&w=730&sz=62&hl=en&start=1&tbnid=P0uKUd4AGlxwRM:&tbnh=142&tbnw=134&prev=/images?q=response+card+turning+technology&gbv=2&hl=en&safe=active&sa=G�


DAU Learner Surveys

• Assessment– No conclusive evidence of improvement on exams in classes where

clickers were used• Level 3 course with higher performing students

• Engagement– Most learners agreed that clickers provided increased involvement– Most learners agreed that clickers helped them stay engaged with the

material and helped them understand

• Satisfaction– Most learners enjoyed using clickers – Most learners agreed that clickers helped them better understand by

catching and correcting mistakes

16

Presenter

Presentation Notes

Comments by Bill Fast: Engagement and satisfaction impressions were gleaned from student surveys taken after the use of CPS clickers during their classes. The learning conclusion was based upon statistical analysis of student grades on exams, in classes taught with and without CPS clickers. Level 3 course with high performing students, small sample size, early in use of CPS, added to existing course (n=~100), exam average at >90%


DAU Learner Survey Questions1. I have enjoyed using the clickers.2. The clickers helped me to learn material presented in the seminars and exercises.3. The clickers helped me pay closer attention to the ideas and concepts presented in the

seminars and exercises.4. The clickers helped me to better understand the seminars and exercises by catching and

correcting my mistakes.5. Using the clickers after a seminar or exercise wrap-up took time that could have been more

productively devoted to the next exercise.6. I enjoyed the increased involvement the clickers provided.7. The clickers increased my level of anxiety and tension since I had to participate whether I

wanted to or not.8. I do not like active learning and thus do not like the clickers.9. The clicker questions helped me in preparing for the exams.10. Immediate on-screen feedback (compared to my clicker response) helped me determine my

level of understanding.11. All exam reviews should include clicker questions.

17

DAU Learner Survey Responses

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

1 2 3 4 5 6 7 8 9 10 11

4 = Strongly Agree

3 = Agree

2 = Disagree

1 = Strongly Disagree

Survey of 148 learners in 7 courses in 3 disciplines indicated results were consistent with other DoD findings18


“Engagement” Comments from Learner Surveys

• The clicker helped to keep me engaged when I may not have been paying as close attention as I should have.

• I loved the clickers! Helped me to focus and be more attentive in class! Also made class more fun!

• The clickers sparked my interest in the subject matter.

• Clickers helped with developing dialogue.

19

Presenter

Presentation Notes

Comments by Bill Fast: These student comments were from PMT 352B Section 09-017 that graduated on 8 April 2009 and Section 10-012 that graduated on 25 February 2010. I was the only facilitator using CPS clickers in Section 09-017, but did use them in four different seminars, four exercise wrap ups, and two exam reviews. In Section 10-012, Matt Bampton and Chuck Augustine also used CPS clickers in their exam reviews (one review each), while I used CPS clickers and the MOBI teacher in two seminars, three exercises wrap ups, and two exam reviews.


“Satisfaction” Comments from Learner Surveys

• It is a good learning tool – enjoyed its use.

• Fun and active. More clicker!

• "Clickers" made interaction great!

• The clickers provide … an anonymous way of checking knowledge. – When it’s a hand raising exercise, some people don’t want to guess

because they are afraid they will be wrong. The clickers eliminate that situation since no one knows what any one person answered.

• Time to reflect - gave my brain a chance to absorb content.

20


Other Comments from Learner Surveys

• Use clickers more often. Have more questions.

• Include more variety of question types than just (A – D).

• Use the clickers earlier to understand which lessons/points to emphasize based on class understanding/knowledge.

• The clickers did not help me learn or understand.

• Clickers useful – but don’t pay too much $$$ for them!

21

Presenter

Presentation Notes


• Integrating into legacy courses

• Integrating into new courseware

• Results of instructor surveys

DAU experience with integrating technology with classroom questions

22


• TODAY - Substitution– Use of audience response system to replace

existing interactions• Test review• Knowledge check• Class opinion

• TOMORROW – Augmentation– Adding new capabilities to existing curriculum

• Ongoing assessment of students• Use of Smart Boards by instructors and students

Integrating into DAU legacy courses

23

Presenter

Presentation Notes

Alvin – Currently, we’re integrating audience response into existing courses. For example, in a mature course such as our intermediate program management class, CPS is used to check knowledge before and after a lesson, to review material before an exam, and to quickly poll class opinion – tasks previously done by orally asking students or counting raised hands. The advantage the technology brings us is greater participation due to its anonymity. For example, if I ask students a question requiring recall of a point in a lesson – say, the contract type preferred after Milestone B – I may have a few murmurs and a few hands raised. Once the correct answer is given, some students may convince themselves they knew the right answer even if they actually didn’t. With an anonymous response system, I can force a response and, based on the answer, have a good idea as to whether or not the class understands that particular point or not. Furthermore, students get real feedback as to whether they recall a particular point – we’ve forced them to answer the question and they are told the correct answer. It’s difficult to convince yourself that you know the answer if you got it wrong! As we become more experienced in the use of this system, we have begun looking at ways to improve current courses through the use of this techniques unique capabilities. Our current system has the capability to record student responses to questions throughout the course. This record can provide us with a method of identifying weaknesses in student learning as the course progresses. This would allow us to intervene with the student to determine if the student is having difficulty with a particular part of the course, with time to help the student before a more critical assessment, such as an exam. Note that proper design of knowledge checks can provide us with an objective indicator of mastery, rather than relying solely on a student’s subjective self-assessment AND their willingness to self-identify. Smart Boards provide another potential method for both collaboration and for new course techniques. Student teams can now interact using the same workspace, allowing new exercises such as highlighting important points in a project plan or collaborating on the design of a product, such as creating an shared set of evaluation criteria. These collaborations can also be stored and interacted with later in the course.


Integrating into new classroom courses

• Examples: Requirements Management • Advanced course

• Today – “Jeopardy” style game used to review content

• Tomorrow – use of knowledge check and facilitation

questions

• Unexpected benefit – course instructors more aware of need

for student engagement, moving away from just lecture

24

Presenter

Presentation Notes


Lean 6-Sigma (LSS) Green Belt Training

• Audience: DoD mid-level supervisors • Duration: 4.5 days, 8 hours per day • Questioning Strategies Employed:

– Course “pretest” about LSS knowledge & confidence to lead team process improvement activities

– Interest & readiness to learn at beginning of each lesson– Subject knowledge at beginning, middle & end of lessons– End of course exam review (Jeopardy style game)

• Comparison of post course (exam) with pre-course (pre-test) demonstrated growth in learner knowledge and confidence

25

Faculty Feedback about “Clickers”

PROs• Forces instructor to think about

questioning• Helps assess audience

understanding • Generates discussion• Enables “shy” students to

engage and get feedback

Cons• CPS won’t fix poorly written

questions• Steep learning curve• Some computers lock up due

to memory limitations• Some believe CPS slows down

the class

Faculty “Pioneers” expressed that clickers are useful for some…but, not necessarily usable with all teaching styles…


Summary

Classroom questioning strategies

Using technology with classroom questions

DAU experience with integrating technology with classroom questions

27

ResourcesBuilding Expertise: Cognitive Methods for Training and Performance Improvement, by Ruth Colvin Clark, Third Edition, (San Francisco, Pfeiffer, 2008)

Classroom Performance Improvement Project: Improving Submarine Training by Increasing Lecture Interactivity by Capitan Roy Harkins, USN, Lieutenant Commander Al Brady, USN, and edited by Lieutenant Commander Rod Wheeler, USN (November 2003)

Classroom Response System Bibliography - Center for Teaching, Vanderbilt University by Derek Bruff, Assistant Director, Vanderbilt Center for Teachinghttp://www.vanderbilt.edu/cft/resources/teaching_resources/technology/crs_biblio.htm

Classroom Response System Classroom Guide - Center for Teaching, Vanderbilt Universityby Derek Bruff, Assistant Director, Vanderbilt Center for Teachinghttp://www.vanderbilt.edu/cft/resources/teaching_resources/technology/crs.htm

Questioning Strategies for Audience Response Systems: How to Use Questions to Maximize Learning, Engagement, and Satisfaction by Will Thalheimer, PhD, Work-Learning Research, Inc. Somerville, MA (March 2007) http://www.work-learning.com/catalog

Teaching with Classroom Response Systems: Creating Active Learning Environments, by Derek Bruff, Vanderbilt University, (San Francisco, Jossey-Bass, 2009)

Teaching with Classroom Response Systems Blog by Derek Bruffhttp://derekbruff.com/teachingwithcrs/

http://www.vanderbilt.edu/cft/resources/teaching_resources/technology/crs_biblio.htm�

http://www.vanderbilt.edu/cft/resources/teaching_resources/technology/crs.htm�

http://www.work-learning.com/catalog�

http://derekbruff.com/teachingwithcrs/�

Resources, cont.The FIT Model of Instruction: Integrating Student Response Systems into Group Collaborative Learning to Enhance Interaction, Comprehension and Retention by Steve Huff, PhD, B&H Consulting Incorporated (2008)

Turning Talk –“A Social Learning Community” for users of the Turning Talk products but also a good resource on using clickers in general. Specific areas for K-12 and Higher Ed practitioners http://www.turning-talk.com/

The Power of Great Questions, Michael Marquart, T+D Magazine, February 2007

Questions? Comments? Want to Learn More? Contact Us!Steve Brown, Logistics Management, [email protected]

Bill Fast, BCEFM, CNE,[email protected]

Deb Fuller, Instructional Systems [email protected]

Dr. Steve Huff, [email protected]

Alvin Lee, PML, [email protected]

http://www.turning-talk.com/�




mailto:[email protected]�











innovations in elearning 2 june 2010elearningsymposium.gmu.edu/2010/presentations 2010... ·...

Documents