etecportfolio.files.wordpress.com€¦ · web viewalpha test of project design: teacher/mentor...

ALPHA TEST OF PROJECT DESIGN 1

Alpha Test of Project Design:

Teacher/Mentor Training- Active Listening

Matthew Atherton

Kristine Holloway

Laura Mitobe

California State University, San Bernardino


Our project was designed to instruct potential teacher-mentees in improving their active

listening skills so that they could work more effectively with new teachers. We chose this project

because it can be difficult to retain new teachers. Wiebke & Bardin noted that lack of

institutional support is a major reason for beginning teachers’ failure to persist in their teaching

career (2009). Albert Bandura’s theory of observational learning led to our use of video and

avatar representations of real-world, classroom situations and interactions between mentors and

mentees. As Greener (2009) noted, Bandura’s theory concludes that learning can be

accomplished through seeing, retaining, and reproducing correct behaviors when the learner is

sufficiently motivated. Our subjects, beginning teachers, are motivated to learn in order to

enhance their effectiveness as educators and to retain their positions.

We chose to embed formative assessments into the course so that our learners could see

their progress in the various learning modules while, at the same time, we would gain knowledge

of the effectiveness of our instructional tool (Vonderwell & Boboc, 2013). The learning needs of

adults were addressed through the brevity of each module, the use of presentation and

demonstration to encourage practice and feedback by the participants, and through providing

constructive feedback at the end of each lesson to reaffirm or correct participant learning

(Alsofyani, Aris,Eynon, & Majid, 2012). Test subjects were chosen because they were

professional educators and had peer-mentoring responsibilities at their work. This allowed them

to judge the training more effectively.

The Analysis, Design, Development, Implementation, and Evaluation or (ADDIE) model

was used to develop our training. In the development phase, as a result of our own assessment,

we created a guide, or table of contents, to allow the user easier access to the course modules.

Multimedia resources that developed the course content were used to reinforce learning


objectives. We were able to gain the input of, and to respond to, the design and content

suggestions of expert collaborators and evaluators of our project in the implementation test phase

(Shibley, Amaral, Shank, & Shibley, 2011). The process of creation, analysis, evaluation, and

revision worked synergistically within the ADDIE framework in the development of this project.

Data Analysis

An ongoing evaluation was conducted throughout the development of the project. The

navigation and content was reviewed and discussed multiple times at each step. The intent of

each review was to create and improve upon a product that was easy to navigate and offered

authentic content for the instruction of learning objectives. Modifications were made as deemed

necessary at each stage of development.

Initial feedback from participants that were not involved in the development occurred

during the videotaping. We solicited verbal evaluations about the content prior to the filming

from our actors who are experts in the field of mentoring.

The formative evaluation of our prototype was sent to respondents who have experience

in the field of education and mentoring of teachers. They were asked to complete the mini-course

and answer questions on a 5-point Likert Scale on Survey Monkey. They had the opportunity to

provide comments for each question and two narrative questions regarding content and

navigation. The testers included teachers who are also experienced mentors.

Our testers were:

● Julette Barta CPhT, MA Ed.-Curriculum Specialist and Lead Mentor Teacher for Colton,

Redlands, Yucaipa Regional Occupational Program (CRY-ROP).

● Stephen Stalker- Teacher/Mentee

● Mike First- Teacher/Mentor


● Mary Rees- Coordinator of Teacher Support, BTSA Coordinator

● Emma Yanez- Teacher, BTSA LeadershipTeam

● Kathleen Fredette- Teacher, BTSA Mentor

● Annette Heins- Teacher, Principal

Our field test revealed that the testers agreed, or strongly agreed with the effectiveness

and quality of the content, text, audio, and video. The weakest area according to our respondents

was navigation. Questions about the ease of the program's use and the intuitiveness of navigation

received the lowest Likert Scale mean score, 3.75 and 2.67, respectively, on a 5-point scale.

Respondents indicated their agreement or disagreement on the following scale.

5-Strongly agreed 4-Agreed 3-Neutral 2-Disagreed 1-Strongly Disagreed

The respondents were asked to relate their level of agreement with the following statements:

● The text could be easily read.

● The audio was easily understood.

● The video was understandable.

● The video was good quality.

● The navigation was intuitive.

● Were the colors distracting.

● Was the program easy to use.

● Was the content beneficial.

● Was the content effective.


Figure 1

Figure 1 shows the mean score for each question on the evaluation. The response choices ranged from 1-5 with 5 indicating the strongest agreement and 1 the strongest disagreement, with 3

being neutral.

Unfortunately, not all of our testers completed the final evaluation due to some confusion

in the navigation of our project. Four of our testers completed the overall evaluation, 6

completed the survey for the paraphrasing module, and 4 completed the survey for the clarifying

module. One respondent could not access the program from her computer, so she went through

the program with Laura Mitobe and provided verbal feedback.

The user was asked to complete each learning module. When a module was completed, a

green check appeared on that module's navigation button, indicating it was complete. When all

three modules were completed, the formative evaluation link appeared, allowing the tester to

respond to the survey and rate the overall product. Some of our respondents completed the

surveys for the paraphrasing and clarifying modules, but did not realize the overall formative


evaluation also needed to be completed. Consequently, we had more feedback on the

paraphrasing survey than the clarifying, or the overall survey.

Figure 2 depicts the percentages of the respondents’ ratings on each of the Likert Scale questions.

Figure 3

Figure 3 shows responses in regard to the effectiveness of the learning activities for paraphrasing.


Figure 4

Figure 4 illustrates the users responses to questions about the learning activities in the clarifying module.

What We Learned

Content

Although there is room for improvement in the content of our materials, overall, the

testers found it satisfactory. One hundred percent of our users strongly agreed, or agreed that the

content was beneficial and effective.

Comments from our testers regarding content:

● “Very simple and to the point. It really illustrates the concepts without lecturing. The

examples clearly show what to do and what not to do.”

● “It is off to a great start! Overall it is very easy to follow and well structured. The content

is well thought-out too. I look forward to the next sequence of this.” Baek

Navigation

We learned that navigation of the program was an issue in several ways:

● The back button was confusing.


● The menu could not be used to navigate through the program because it would not

register completion of each module.

● If completion was not registered, the user could not access the assessments.

Figure 5

This screen exemplifies how the green arrows indicate completion of the modules, then allow the link for the evaluation to be accessed. If the modules are not completed, the

evaluation cannot be completed.

Audio was provided to comply with ADA requirements. However, the testers preferred to

have the option of turning the audio ofUser comment:

● “It is confusing to have to go back. There are no instructions. Without instructions, there

is no way to know the green arrow, or change of color on the menu indicates

completion.”


Strengths

Most users felt the videos were effective and they liked the captions. The content was a strength

and users felt it was beneficial and effective. User comments:

● “Liked the closed captioning.”

● “I think you have a good start. I like how it goes from video to mouse over, etc. I think

the short assessments are good as well. It's hard to get acting that looks real, so I think

they did an ok job given what you are trying to accomplish. I like that things are short

and to the point.”

● “I was surprisingly interested in the material and happy with product. I usually am not.”

Weaknesses

Our weaknesses mainly involved some elements of navigation. Users had to click back

buttons to return to the main menu. The evaluation survey was hidden, so the user did not

anticipate that there was an additional set of questions to answer.

Most users wanted the option of turning the audio off. When the user was asked to select

options for clarifying and paraphrasing, they had to wait for the narrator to finish reading or the

audio double played.

● “It was difficult to get to the evaluation without instructions. It should show in the menu

so the user knows it is there.”

Some users suggested more examples and an explanation as to why the paraphrasing and

clarifying answers were correct.

● “Explain why the answer is correct. Give a 2nd chance to choose correct answer.”

Revisions


Revisions should include easier access to the assessments. The respondents were pretty

clear that they would like an option for turning off the audio.

“Would like to turn audio off.”

● Navigation should be more linear without having to use the back button to access the next

module.

Allow the option of navigating through the menu, as well as links to the next page.

● “I used the menu to navigate, and the assignments weren't clicked off. I should be told to

use the main portion, or have the menu click sections off.”

Allow the user to progress to the next screen and choose options without waiting for the text to

be read by the narrator.

● This is a suggestion for a future, real-world example of paraphrasing:

T. “I would like to use assessment to improve my students grades.”

M. “You would like to use assessment to adjust your teaching to help your students

master the objectives.”

Ineffective response: “You should give your students tests to assess their levels and find

the weaknesses in your teaching.”

● “I'd create more activities or examples. What you have here is pretty nice but it would be

nice to have more examples.”

Conclusion

Overall, our project had positive reviews as indicated by the surveys and comments.

There are areas of navigation that could be improved:


Use a more linear progression through the program. Avoid back buttons for a return to the

introductory screen. If the project were to be expanded, more examples should be added the user

should have the option of turning the audio off.


References

Alsofyani, M., Aris, B., Eynon, R., & Majid, N. (2012). A preliminary evaluation of short

blended online training workshop for TPACK development using technology acceptance

model. Turkish Online Journal of Educational Technology - TOJET, 11(3), 20-32.

Greener, S. (2009). e-modeling--helping learners to develop sound e-learning behaviours.

Electronic Journal of E-Learning, 7(3), 265-272.

Shibley, I., Amaral, K. E., Shank, J. D., & Shibley, L. R. (2011). Designing a blended course:

Using ADDIE to guide instructional design. Journal of College Science Teaching, 40(6),

80-85.

Vonderwell, S., & Boboc, M. (2013). Promoting formative assessment in online teaching and

learning. Techtrends: Linking Research & Practice to Improve Learning, 57(4), 22-27.

Wiebke, K., & Bardin, J. (2009). New teacher support: A comprehensive induction program

can increase teacher retention and improve performance. Journal of Staff Development,

30(1), 34-36,


Appendix

Overall Formative Evaluation:


Paraphrasing Formative Evaluation:


Clarifying Formative Evaluation:

etecportfolio.files.wordpress.com€¦ · web viewalpha test of project design: teacher/mentor...

Documents