INSTRUCTIONAL EVALUATION: “WORDPRESS ESSENTIAL TRAINING” BY LYNDA.COM USED AS AN INTERNAL STAFF DEVELOPMENT TOOL

Evaluation Plan – EDCI 577 Sec 004

Jason Mutzfeld

Training Overview

Faculty of the university department of education will complete a self-paced training 5-hour training entitled “WordPress Essential Training” online through Lynda.com so they will be able to create a personal web presence using the WordPress platform.

The website Lynda.com is an online repository of video training classes that is part of LinkedIn social network platform. Members of the site have access to a wide variety of topics most related to technology and design fields. Their course content ranges from beginner to advanced and are open to any subscriber. Use of Lynda.com’s materials in corporate and organizational environments is analogous to sending an individual to a training class offsite or using a pre-prepared training program in-house.

The course that is used in this evaluation is “WordPress Essential Training” which provides a general overview and instruction in how to use the WordPress Content Management System (CMS) to construct websites. This course is available online through any modern browser and consists a series of instructor led videos and screen captures comprising 14 chapters totaling 5 hours and 18 minutes of instruction. The videos are designed as an introduction to WordPress as a platform to create a website. Each user is assigned a Lynda as part of the departmental license agreement. Videos are self-paced and can be completed anywhere there is web access at the trainee’s convenience.

After completing the training online, each user will be given access to a WordPress site of their own to use with their students and tasked to build a site that they will be asked to update with current content and integrate into their instruction.

Context and Target Audience:

In this scenario, the Education department of a small university would like to have all its faculty become proficient in WordPress, a content management system used in the creation of websites. Each faculty member will be given access through the university to Lynda.com, an online training site, and tasked with completing the course “WordPress Essential Training” within a two-week time frame. At the end of the course the faculty member will be able to add the completion of the training to their LinkedIn professional skills profile.

The department chair would also like each faculty to utilize WordPress as part of their online presence and increase the use of digital media so they can better connect with students. To this end, one of the main reasons for requiring this class is that can apply what they learn directly to create their own individual WordPress site. To accomplish that they will be required to demonstrate proficiency in the product by creating a sample site of their own design after completing the course. The Academic Technology staff will be providing support in this part of the process.

Executive Summary

Evaluation Objective:The primary objective of this training evaluation is to determine if training faculty to both use and implement a WordPress-based website for would improve the educational experience for their students. The belief that as current students are more familiar with online digital tools and resources that each faculty member having their own presence online related to their classes will provide another method of communication for them to connect with their students.

EDCI 577 004 Fall 2016 1

Design of Evaluation and Rationale:

Following the Kirkpatrick model (Kirkpatrick 2006) evaluation will start by the trainee’s satisfaction with the course and then proceed to measure if the trainee successfully learned from the course. It is important that the trainee find value in this training and that they can provide evidence of being able to apply what was learned in the course. To begin, we will start with a survey of the learners prior to the training and we will follow with a review of each trainee’s final project. We will continue to monitor the trainee’s individual use of WordPress in their workday to determine if the product is being used. To justify the value of this training program and its continued use, we will examine if the implementation of this training results in a positive reaction from both faculty and students with a focus on answering the question: “Does the faculty’s use of WordPress improve the education experience for students?”

Findings:

This training is scheduled to begin at the start of the calendar year and evaluation will proceed over the course of the Spring Semester with results to presented to chief stakeholders in May. If the program is successful it will be asked to be added to departmental budget and be mandatory for all new hires.

The key data elements to watch for during this evaluation are:

Instructor’s perceived technical ability Instructor’s attitudes towards this project as a whole The frequency of updates to their site to gauge usage Student responses to the use of the site

Should all four of those elements be found to be successful then the program may well be judged to be a success.

Recommendations:

Since this training program is a pilot program and has not been tested there are many possible variables to consider. There is a high probability that faculty may not desire to implement this program and will resist it even to the point of not participating. Without strong support from the Vice-President of Academic Affairs, the Department Chair, and other senior leadership there is high chance of this producing sub-standard results that would reflect poorly on the department as whole. Launch of the training program should include a persuasive argument as to the value of the both the training and the new web presence initiative with an intention to foster both an individual desire to achieve and belief there is intrinsic value in participation.

It is also possible that this level of technical skills is beyond some of the faculty and results could show that they need additional support to effectively integrate their web presence into their instructional methods.

EDCI 577 004 Fall 2016 2

Instruments and Procedures:

Measuring Reaction & Satisfaction (Kirkpatrick Level 1)

This level will consist of two surveys: a pre-class and post-class assessment. The first survey is to gauge the attitudes of the trainees to the subject and the process itself. The second survey is to solicit feedback on the process as a whole to see how it can be improved or if an alternative method of instruction needs to be found. These surveys will be required of all trainees and completed online. Results will be shared with the participants once assessment is complete. Both the pre-class and post-class survey will consist of 7 questions and use a 5-point scale. The scores will be averaged on each and increases/decreases. Surveys will be written plainly and are designed to give the trainee a perceived voice in the process.

See Appendix A for the pre-class survey and Appendix B for the post-class survey.

Measuring Learning (Kirkpatrick Level 2)

Since the skills we are seeking in this training are practical in nature, the second level will be a review of the final product each trainee will need to complete. Once a trainee has completed their training they will receive a logon to their WordPress account plus a guideline document and then they will be given an additional two weeks to create a personal website that will enhance their professional profile and improve communications.

After they have completed this assignment, the pages will be reviewed and assessed based on the criteria trainees were given. Successfully completing the assignment will entail having a WordPress site containing the required elements listed in the criteria given. This portion of the evaluation will be shown to be successful when the majority of the trainees have sites to publish to the general public. A score will be assigned to each user based completing 10 tasks (10 points per = 100 points).

See Appendix C for the assessment criteria.Measuring Behavior (Kirkpatrick Level 3)

Providing the training and WordPress sites to each faculty member is only the first step. Content creation for each’s faculty member’s site would ideal become a habitual change and their sites would become a dynamic tool for connection with students, not just a static informational source. To this end, it will be important to monitor the participation of each faculty member to encourage them to move beyond participation in this project but to adopt the tool and update it frequently. A survey will occur four, six, and eight weeks after training completion to determine if trainees are updating their site. This will be done by the Academic Technology department by examining each trainee’s WordPress site. Each trainee that has not updated their site will be flagged for contact by the Academic Technologist to determine if they need assistance or have questions.

See Appendix D for the assessment criteria.

Measuring Results (Kirkpatrick Level 4)

The ultimate goal of this projects will be for faculty to use WordPress as a tool to enhance the education experience of students, so to measure it we will be focusing on students. Eight weeks after the training is complete students will be polled to determine their opinions of the WordPress sited used by their

EDCI 577 004 Fall 2016 3

instructors. Responses will be tabulated in percentages of positive answers by students to four questions relating to their perceptions of the sites and their use by the faculty.

After this final survey, the Academic Technology department will compile a report of the assessments done at all for levels which will be presented to the Dean of the school in addition to the Vice President of Academics. The focus of this presentation will not only show the success of the program and opportunities for improvement, it will be used to value of investing funds in this training in the future.

See Appendix E for the assessment criteria.

Data Collection and Reporting:

Level 1 pre-class results and post-class results will include several questions that are used to determine if there is a shift in attitudes from beginning to end. The goal is to prove that this training promotes a feeling of increased competence, efficacy, and that trainees will perceive value in the process. The results will be averaged and presented in the departmental quarterly report.

Level 2 results will be the reported by averaging the scores of each user both based on the number who have completed the practical project and then again adding the scores of those who did not. This second average may serve to lower the overall median score however it is necessary to get an accurate picture of participation. Both results will be shown in a three bell curves; one for each data set and a third with them overlapped. The hoped for outcome would be a mean score over 7.5.

Level 3 Results will be measured by reviewing each faculty member’s site and determining if it being updated on a regular basis. Each of criteria to be reviewed will be assigned a point score and each faculty member will be assigned score at each point of review. At the end of the three reviews an average score will be assigned to each faculty member and recorded.

Level 4 student survey results are more qualitative than quantitative in nature however it is important to establish a connection between them and the results of the other levels because they most directly answer the question “Does the faculty’s use of WordPress improve the education experience for students?”

The ultimate Level 4 evaluation is found in compiling the results of the evaluation assessments in to a report to the stakeholders. The resulting presentation the Level 4 evaluation will comprise the following elements:

1. Review the scope and intention of the project2. Show an increase in faculty participation and satisfaction through changes in pre- and post-

survey results for Instrument #13. Show an increase in faculty’s skill with WordPress having completed training and practical

implementation phase with results from instrument #24. Show that faculty are using the new WordPress site by discussing trends of use found in the

results of Instrument #3 over the three measuring periods5. Answering the question “Does the faculty’s use of WordPress improve the education experience

for students?” with a summary of the results of Instrument #4 averaging of the responses to the questions on Instrument #4 (e.g. “80% of the students responded ‘yes’ to when asked if they had visited the instructor’s new webpage ‘)

EDCI 577 004 Fall 2016 4

6. Present suggested changes to the program that could improve results in future training7. Make a persuasive case for the expenditure being placed into the budget for next year based on

previous points

Project Plan and Timeline

The training is slated to take place asynchronously over a two-week period and evaluation of use will be done over two-month period following completion. A Gantt chart and projected timeline has been included in the appendices.

See Appendix F for the Timeline

References:

Kirkpatrick, D. & Kirkpatrick, J. (2006). Evaluating training programs: the four levels (3 rd ed.). San Francisco, CA: Berret-Koehler Publishers, Inc.

EDCI 577 004 Fall 2016 5

Appendix A – Instrument #1A

Thank you for participating in this development course in using WordPress! We hope that as you go forward you not only

Please be sure to include the code sent with your welcome email when completing this form. All results are anonymous and the code simply indicates you have completed this portion of the training.

Good luck and please feel free to contact the Academic Technology Department with any questions during this process!

Rate your ability with online digital tools A in general: _________

1. Expert - I “live online” and can teach others2. Semi-Expert – I can get by with minimal assistance 3. Professional - I can get by with assistance but am not as confident as some4. Semi-Uncomfortable – I most don’t enjoy working online and need help frequently5. Techno-phobic – I detest digital tools

Rate your personal level of ability of with creating online content B: _________

1. Expert – I frequently create online content for my web presence.2. Semi-Expert – I can create online content if necessary3. Professional – if the job requires, I can do it4. Semi-Uncomfortable – I avoid doing it if I can help it5. Techno-phobic – I don’t ever create content and do not enjoy it

Rate your level of overall level of familiarity in creating websites: _________

1. Expert – I have created some prior to this2. Semi-Expert – I have worked on some, not many but can figure it out3. Professional – With the proper instruction I believe I could do it4. Semi-Uncomfortable – I could do it but would need much help5. Techno-phobic – I don’t ever see myself creating a website

How familiar are you with WordPress C ? _________

1. Expert – I use WordPress currently and frequently 2. Semi-Expert – I have used it before but not frequently3. Professional – I can if I have to once I am trained4. Semi-Uncomfortable – I would be uncomfortable with it but could with help5. Techno-phobic – I would be too uncertain of my skill to want to use it

How comfortable are you with training online? _________

1. Expert – I have done many online courses in the past2. Semi-Expert – I have worked on some, not many but can figure it out3. Professional – With the proper instruction I believe I could do it

EDCI 577 004 Fall 2016 6

4. Semi-Uncomfortable – I could do it but would need much help5. Techno-phobic – I don’t ever see myself creating a website

Do you believe there is value in a self-curated D online web presence E? _________

1. Expert – I am easy to find online and it says what I want it to say2. Semi-Expert – I have some online presence, but not much3. Professional – You can find the basics about me online4. Semi-Uncomfortable – I worry about privacy and don’t like sharing online5. Techno-phobic – I refuse to share things online unless required to

Are you looking forward to this class? _________

A) Yes! Very ExcitedB) Yes, but nervousC) No, this a waste of my time

Signature Code: _____________________________

Key terms:A) Digital Tools: examples include Blackboard, blogging, YouTube, Social MediaB) Online Content: creating digital photos, memes, blog postsC) WordPress is a Content Management System used for creating blogs and webpagesD) Curation is the process of creating and management web content such as pictures and blog

postsE) Web Presence: Your personal profile online, such as a website, that tells people about you.

EDCI 577 004 Fall 2016 7

Appendix B – Instrument #1B

Congratulations you’ve successfully completed Part 1 of your WordPress training!

Please take a moment to share your thoughts with us on your experience thus far so we can continue to improve instruction in the future

Do you believe this course will help you create a WordPress site? _________

1. Yes, I am confident it will2. Yes, but I will have some things I will help on3. Unsure, but with help I might be able to4. No, I’m going to struggle to complete the next step5. No, I’m didn’t learn anything of value

After this instruction, rate your personal level of ability of with creating online content: _________

1. Extremely comfortable2. Comfortable3. Undecided4. Moderately uncomfortable5. Very uncomfortable

Did this course increase your familiarity in creating websites? _________1. Yes, I am confident it will2. Yes, but I will have some things I will help on3. Unsure, but with help I might be able to4. No, I’m going to struggle to complete the next step5. No, I’m didn’t learn anything of value

Are you now more familiar with WordPress than you were before? _________

1. Yes, I learned learned a lot2. Yes, learned a fair amount3. I learned some but am still not fully confident I know how to use it without more help4. No, I’m going to struggle to complete the next step5. No, I’m didn’t learn anything of value

What is your opinion of training online from Lynda.com after this portion of the class? _________

1. Online learning from Lynda.com is valuable and effective2. Online learning from Lynda.com is somewhat effective3. I am neither impressed nor unsatisfied with Lynda.com4. Online learning from Lynda.com is not as valuable or effective as I’d hoped5. Online learning from Lynda.com is not valuable or effective

Do you believe there is value in having your own web presence? _________

EDCI 577 004 Fall 2016 8

1. Yes, I am convinced there is value 2. Yes, but I have some doubts3. Unsure, I am not convinced but willing to try 4. No, I’m not convinced5. No, I am not convinced and this project has no value

Did you enjoy this portion of the class? _________

A) Yes, very much!B) Somewhat, neither pleasant or unpleasantC) No, very boring

Signature Code: _____________________________

EDCI 577 004 Fall 2016 9

Appendix C – Instrument #2

User: ___________________________________________

Reviewer: ___________________________________________

Guidelines for Personal Web Presence using WordPress. This should be completed 10 working days (2 weeks) after the completion of the video training.

A value of 10 points will be assigned to each of item totaling 100 points. Values are either 0 for not completed, 5 for partially completed, and 10 for completed.

Please check all that apply:

1. Did they review and sign the university Acceptable Use policy and return it via email to the

department administrative assistant? _________

2. User was able to successfully log onto their WordPress site _________

3. Did they change their password? _________

4. Did they upload their professional headshot? _________

5. Did they add the university logo and contact info? _________

6. Did they create a biography? _________

7. Did they create a ‘Connect with me’ page with email? _________

8. Did they link to the departmental password? _________

9. Did they change the theme or color scheme? _________

10. Did they add the Academic Technologist as a secondary administrator? _________

Total: __________

Progress Scale:

Above Satisfactory = 75 - 100 pointsSatisfactory = 60 – 75 pointsBelow Satisfactory = 40 – 60 pointsUnsatisfactory = Less than 40 points

EDCI 577 004 Fall 2016 10

Appendix D – Instrument #3

WordPress Engagement Review

Date of Review: Circle one: Week 4 / Week 6 / Week 8

Name: ______________________________________________________

Subdomain: __________________________________________________

Creation date:

Date of last update:

Number of updates since last review:

Last date of contact:

Flag for contact? Y / N Support ticket created for follow-up? Y / N

Consult the previous review* the user and answer the following questions:

Has the user added written content to their WordPress site? Y / N (4 points)

Has the user added Audio/Video or Image content to their site? Y / N (2 points)

Is the user directing students to communicate with them on the WP site? (4 points)

Have students engaged with the faculty member on the WP site? (2 points)

Total Score: ___________

* If applicable; Week 4 review will act as baseline score from completion of training class.

EDCI 577 004 Fall 2016 11

Appendix E – Instrument #4

Please take a moment to answer the following four questions about your experience with the department’s recent implementation of WordPress for faculty and instructors.

You will receive a survey for each class in the department.

Your input is extremely important to this project. Thank you for your participation.

Student name: %mail_merge_input%Instructor name: %mail_merge_input%

Question 1) Have you been invited by your instructor to visit their new webpage? Yes / No

Question 2) Have you visited your instructor’s new webpage? Yes / No

Question 3) Do you like the instructor’s page? Yes / No

Question 4) Does the instructor update the site regularly? Yes / No

Question 5) Do you think the site makes class experience better? Yes / No

Bonus for you! Print out the receipt email for completing this survey and deliver it to the department office during normal hours you’ll receive a free candy bar for your efforts.

1.

EDCI 577 004 Fall 2016 12

Appendix F – Timeline & Chart

Proposed Schedule 2017

Timeline

Task Start Date Assigned toTime

(Days)Project Launch Meeting 1/11/17 Project Team 1Lvl. 1 Pre-Assessment 1/18/17 Faculty 7Training session 1/25/17 Faculty 14User Site Implementation 1/26/17 Faculty 14Lvl.2 Site check 1/27/17 Acad. Tech 7Lvl. 1 Post-Assessment 1/27/17 Faculty 7Lvl. 3 Assessment 1 1/28/17 Acad. Tech 14Lvl. 3 Assessment 2 1/29/17 Acad. Tech 14Lvl. 3 Assessment 3 1/30/17 Acad. Tech 14Lvl. 4 Opinion Survey 1/31/17 Students 7Compile Analysis 2/1/17 Project Team 14Create Final Report 2/2/17 Project Team 21Presentation of results 2/3/17 project Team 1

EDCI 577 004 Fall 2016 13