esu revised version 6/15/10 instructional … · 3 section i —context complete ... integrating...

1

KSDE Revised 6-15-10

Program Report Format Program Annual Report for 2007-2010

ESU Revised Version 6/15/10

INSTRUCTIONAL DESIGN

AND TECHNOLOGY

Kansas State Department of Education

COVER SHEET

Institution: Emporia State University

Accredited By: X KSDE X NCATE

Date Submitted: 8-1-10

Name of Preparer(s): Marcus D. Childress

Unit Head Name: Phillip Bennett

Unit Head Phone Number: 620-341-5367 Unit Head Email: [email protected]

Level of the Program: Initial Advanced

Grade levels for which candidates are being prepared:

K-6 5-9 6-12 K-12

Is this program being offered at more than one site? Yes No

If yes, please list the sites at which the program is offered: Online

Program Report Status:

New Program Continued Program Dormant Program

(NEW PROGRAMS MUST SUBMIT SYLLABI)

A PROGRAM WILL NOT BE RECOMMENDED FOR FULL APPROVAL IF IT

MEETS FEWER THAN 75% OF THE STANDARDS.

2

GENERAL DIRECTIONS

The following directions are designed to assist institutions as they complete this program report. To

complete the report, institutions must provide data from multiple assessments that, taken as a whole, will

demonstrate candidate mastery of the Kansas standards. These data will also be used to answer the

following questions. Reviewers expect these prompts to be answered by the report.

Have candidates mastered the necessary knowledge for the subjects they will teach or the jobs they

will perform?

Do candidates meet state licensure requirements?

Do candidates understand teaching and learning and can they plan their teaching?

Can candidates apply their knowledge in classrooms and schools?

Are candidates effective in promoting student learning?

To that end, the program report form includes the following sections:

I. Contextual Information – provides the opportunity for institutions to present general information

to help reviewers understand the program.

II. Assessments and Related Data – provides the opportunity for institutions to submit

multiple assessments, scoring guides or criteria, and assessment data as evidence that standards are

being met.

III. Standards Assessment Chart – provides the opportunity for institutions to indicate which of the

assessments are being used to determine if candidates meet program standards.

IV. Evidence for Meeting Standards – provides the opportunity for institutions to discuss the

assessments and assessment data in terms of standards.

V. Use of Assessment Results to Improve Candidate and Program Performance – provides the

opportunity for institutions to indicate how faculty are using the data from assessments to improve

candidate performance and the program, as it relates to content knowledge; pedagogical and

professional knowledge, and skills; and effects on student learning.

Page limits are specified for each of the narrative responses required in Sections IV and V of the report,

with each page approximately equivalent to one text page of single-spaced, 12-point type. Each

attachment required in Sections I and IV of the report should be kept to a maximum of five text pages.

Although attachments longer than five pages will be accepted electronically, staff will require institutions

to revise reports submitted with lengthy attachments.

Except for the required attachments, institutional responses can be entered directly onto the form.

Specific directions are included at the beginning of each section.

3

SECTION I—CONTEXT Complete the following contextual information:

1. A program of study that outlines the courses and experiences required for all candidates to

complete the program. The program of study must include course titles and hours of credit per course.

(This information may be provided as an attachment from the college catalog or as a student

advisement sheet-- maximum of five text pages.) NEW PROGRAMS MUST SUBMIT SYLLABI

IN THE DOCUMENT WAREHOUSE AND IN A FOLDER ON THE CD.

2. Chart with the number of candidates and completers. (Title-Chart with Candidate Information)1

Limit of 6 pages, not including the charts.

1. Program of Study:

Provide the following contextual information:

Description of the relationship of the program to the unit’s conceptual framework.

The mission of The Teachers College, the school personnel preparation unit of Emporia State

University, is to develop professionals who are critical thinkers, creative planners, and effective

practitioners. Our graduates (graduates of the MS in IDT program) are skilled practitioners who

are prepared with essential knowledge, skills, and dispositions in their fields of specialization.

Candidate learning reflects historical and contemporary knowledge, research, theory, and

practice that meet the academic, personal, and social needs of their students. The vision of The

Teachers College (the IDT department) and personnel preparation unit is to prepare quality

professionals (IDT candidates) who can positively impact the education profession (instructional

design & technology profession) and improve the learning of PK-12 students (and adults in the

workplace and higher education). Professional programs (the IDT program) are designed to

reflect the current knowledge base and effective practices. Curricular coherence is strengthened

through faculty study and dialogue on purpose, course content, and intended candidate learning

outcomes.

Indication of the program’s unique set of program assessments and their relationship of

the program’s assessments to the unit’s assessment system.2

The seven standards for the MS in IDT program were derived by the IDT faculty from the AECT

standards. These standards are concerned with candidate performances, curriculum, faculty,

specialized support services, and evaluation procedures in the area of educational

communications and instructional technologies (ECIT). Curricula and candidate performances

for the advanced preparation of personnel in the field of educational communications and

instructional technologies (ECIT) are grounded in the knowledge base of the field. The domains

of the field include design, development, utilization, management, and evaluation.

Description of the criteria for admission, retention, and exit from the program, including

required GPAs and minimum grade requirements for the content courses accepted by the

program.

The IDT program has a comprehensive system for assessment of candidate knowledge, skills,

and performances. Candidate decision points are shown in the table below. The sequence of the

4

Unit’s system for performance assessment of advanced candidates includes admission to

advanced study at the University and four decision points (left column). Corresponding with the

Unit’s decision points, the sequence of IDT’s system for performance assessment includes

admission to advanced study at the University and 5 Checkpoints (right column).

Assessment of Candidate Performance Table

Sequence of Advanced Candidate

Decisions

IDT Sequence of Advanced

Candidate Checkpoints

Admission to Advanced Study Admission to Advanced Study

Decision Point 1 –

Admission to Program of Study

IDT Checkpoint 1 –

Admission to IDT Program of Study

GPA of 2.75 or more based on the

4-point scale in the last 60 semester

hours of undergraduate study

Resume (2 or more pages)

Two current recommendations via

forms or letters

Access to the Internet and possess a

multimedia capable computer

Writing Competency -A 500 to 700

word formal paper describing your

experiences and career goals

involving instructional design and

technology and your interest in

completing an online degree.


Admission to Field Experience Activity:

internship, practicum, action

research or other approved PK-12

school related activity


Completion of Required Majority of

Courses and Admission to IDT Master’s

Project


Completion of Field Experience

Activity: internship, practicum,

action research or other approved

PK-12 school related activity


Completion IDT Master’s Project

IT899 Online Components


Program Completion IDT Checkpoint 4 –

Student Exit Interview


Final Presentation/Paper

Description of the field and clinical experiences required for the program, including the

number of hours for early field experiences and the number of hours/weeks for student

teaching or internships.

5

In the capstone master’s project experience, our candidates demonstrate their knowledge and

application of instructional design principles, teaching/learning theory, and technology

integration. Candidates must determine an instructional problem within their school or

community and create a solution based upon the systematic design of instruction that includes

analyzing the learners/users, designing the instruction, developing the instruction, implementing

the instruction, and evaluating the instruction. The candidate must apply the principles of

instructional design, use multimedia systems, demonstrate mastery of a variety of software

packages, demonstrate good presentation skills, and apply sound principles of

pedagogy/andragogy. The project typically addresses teaching/learning problems and

procedures encountered while working with students in a P-12 classroom setting, or while

working with adults in a business/industry setting. The project is conceptualized at the beginning

of the candidate’s program, approved by his/her advisor, updated, and refined as the candidate

completes class work during his/her course of study. The final project forms a coherent package

integrating the candidate’s instructional technology educational experiences and his/her

anticipated or ongoing professional responsibilities.

1 KSDE uses the Title II definition for program completers. Program completers are persons who have met

all the requirements of a state-approved teacher preparation program. Program completers include all

those who are documented as having met such requirements. Documentation may take the form of a

degree, institutional certificate, program credential, transcript, or other written proof of having met the

program’s requirements. 2This response should clarify how the key assessments used in the program are derived from or informed by

the assessment system that the unit will address under KSDE/NCATE Standard 2.

6

2. Chart with Candidate Information:

Directions: Provide three years of data on candidates enrolled in the program and completing the

program, beginning with the most recent academic year for which numbers have been tabulated. Please

report the data separately for the levels/tracks (e.g., baccalaureate, post-baccalaureate, alternate routes,

master’s, doctorate) being addressed in this report.

Program (initial ): (insert name of program)

Academic

Year

# of Candidates Enrolled

in the Program3

# of Program

Completers4

2007-2008

2008-2009

2009-2010

Program (alternative): ( insert name of program)

Academic

Year

# of Candidates Enrolled

in the Program

# of Program

Completers

2007-2008

2008-2009

2009-2010

Program (Post-baccalaureate – Added Endorsement): MS Instructional Design

and Technology

Academic Year # of Candidates

Enrolled in the

Program

# of Program

Completers

Master’s/Ed.

Specialist/Doctoral

2007-2008 195 37 37

2008-2009 202 40 40

2009-2010 223 50 50

3An enrolled candidate is officially admitted to the program.

4KSDE uses the Title II definition for program completers. Program completers are persons who have met

all the requirements of a state-approved teacher preparation program. Program completers include all

those who are documented as having met such requirements. Documentation may take the form of a

degree, institutional certificate, program credential, transcript, or other written proof of having met the

program’s requirements.

7

SECTION II— ASSESSMENTS AND RELATED DATA

In this section, list the multiple assessments that are being submitted as evidence for meeting the Kansas standards for this content area. All

programs must provide a minimum of six assessments, maximum of eight assessments; assessments #1-6 are required for all programs. For each

assessment, indicate the type or form of the assessment and when it is required/administered in the program.

Include an updated Assessments Chart in the format shown below.

Name of Assessment5

Type or form of

Assessment6

When the Assessment Is Required/

Administered7

1. [Licensure assessment, or other content-based

assessment]8

* (Required)

a. Praxis II

b. PLT

N/A N/A

2. Instructional Design Final Project [Assessment of candidate ability to plan instruction]

* (Required)

Project IT800

3. MS Project Online Component

[Assessment of clinical experience]9

* (Required)

Project IT899

4. MS Project Presentation/Paper [Assessment of candidate effect on student learning]

* (Required)

Presentation/Paper IT899

*Required Assessments

5Identify assessment by title used in the program; refer to Section IV for further information on appropriate assessment to include.

6Identify the type of assessment (e.g., essay, case study, project, comprehensive exam, reflection, portfolio).

7 Indicate the point in the program when the assessment is administered (e.g., admission to the program, admission to student teaching/internship,

required courses [specify course title and number], or completion of the program). 8Assessment #1a Praxis II sub-score data may be used as an assessment for meeting content standards. A data table for Praxis II content test and a data

table for sub-score data must be submitted but a rubric is not required 9 Clinical experience includes practica, student teaching and internships.

8

5. Multimedia Project [Content-based

assessment] Examples of assessments include

comprehensive examinations, projects,

comprehensive portfolio tasks and score/s aligned to

standards OR course grades-based assessments10

related to content knowledge.

Project IT810

6. Web-Based Design Project [Content-based

assessment (Required)] Examples of assessments

include comprehensive standard examinations, case

studies involving many content standards, projects,

comprehensive portfolio tasks and score/s, and

follow-up studies related to content knowledge.

Project IT820

7. Issues in Distance Education Final Project [Additional assessment that addresses Kansas

content standards ]

Project IT830

8. [Additional assessment that addresses Kansas

content standards ]

*Required Assessments

10. Course grades-based assessments can only be used for Assessment 5.

SECTION III—STANDARDS ASSESSMENT CHART

9

For each Kansas standard on the chart below, identify the assessment(s) in Section II that address each standard. One assessment may

apply to multiple Kansas standards. In Section IV you will describe these assessments in greater detail and summarize and analyze

candidate results to document that a majority of your candidates are meeting Kansas standards. To save space, the knowledge and

performance indicators of the Kansas standards are not identified here, but are available at – www.ksde.org . The full set of standards

provides more specific information about what should be assessed. Please include information on assessments used for PreK if this is

an all-level program.

MS in Instructional Design and Technology Program Standards

APPLICABLE

ASSESSMENTS FROM

SECTION II

Standard 1 Evaluate and use computer and related technologies to support the instructional process. 2, 7

Standard 2 Apply current instructional principles, research, and appropriate assessment practices to

the use of computers and related technologies. 2, 3

Standard 3 Demonstrate knowledge of computer usage for problem solving, data collection,

information management, communications, presentations, and decision-making utilizing multimedia,

hypermedia and telecommunications technologies.

5, 6

Standard 4 Design and develop student learning activities that integrate computing and technology

for a variety of student grouping strategies and for diverse student populations. 2, 6

Standard 5 Demonstrate knowledge of multimedia, hypermedia, distance education and online

learning theories and practices to support instruction. 5, 6, 7

Standard 6 Possess functional knowledge of practices, theories, major research findings and trends

related to the field of instructional design and technology. 3, 4, 7

Standard 7 Possess functional knowledge of currently accepted principles of design and

implementation as they relate to careers in the field of instructional design and technology 2, 3, 4

http://www.ksde.org/

10

SECTION IV—EVIDENCE FOR MEETING STANDARDS

DIRECTIONS: Information on the multiple assessments listed in Section II and the data findings must be reported in this section. The

assessments must be those that all candidates in the program are required to complete and should be used by the program to determine candidate

proficiencies as expected in the program standards.

For each assessment, the evidence for meeting standards should include the following information:

A brief description of the assessment, project, portfolio and its use in the program. Explain specificity of the assessment to the standard/s. An

assessment may assess several standards at the same time;

The alignment of the assessment with the specific KSDE standards addressed by the assessment, as they are identified in Section III;

A brief summary of the data findings;

An interpretation of how that data provides evidence for meeting standards.

The response to each assessment is limited to the equivalent of two text pages.

For each assessment listed, you will need to attach the following:

Scoring guides, criteria or rubric (specific to content of standard/s) used to score candidate responses on the assessment;

A table (include # of candidates) with the aggregated results of the assessment providing, where possible, data for at least the most recent

three years. Data should be organized according to the categories used in the rubrics/scoring guide/criteria. Provide the percentage of

candidates achieving at each category.

For each assessment #1a (sub-score data) and assessment #5 (course grades-based assessment), you will include the following information:

Praxis II sub-score data tables must be clearly labeled to indicate alignment with the standard it is assessing. Section IV narrative must

clearly show alignment of sub-score data to the standard or elements of the standard.

Course grades-based assessments have a brief description in the matrix. A more detailed and specific discussion of the alignment of

activities, exams, and project in the course to the standard should be included in the narrative description of assessment 5. The course

grades-based assessments data tables will be included in the narrative of assessment 5. Each course grades-based assessment is numbered

and lettered as 5A, 5B, 5C, 5D, 5E and 5F. Use the same number and letter in the narrative and the data table. If the course grades-based

assessments are used as evidence for meeting two standards, the course key assessments’ data (exams, projects, portfolio tasks) must be

disaggregated in a data table for each of the two standards. This is necessary to provide evidence of meeting each standard. One course

MAY NOT MEET more than two standards.

In the two columns for attachments, click in the box for each attachment to be included with the report. Each attachment should be no longer than

five pages. The two attachments related to each assessment must be included for the program report to be complete. The report will not be

reviewed until it is complete.

11

Assessment #1: (Required) CONTENT KNOWLEDGE: Data from licensure tests for content knowledge. Provide assessment information

as outlined in the directions for Section IV. PRAXIS II content and PLT (If applicable, scores will be provided by The Teachers College).

Submit overall score for all candidates. Data tables for standards must be PRAXIS II sub-score data that are aligned to specific

standards/s. Limit of 2 pages excluding attachments.

Attachments

Assessment #1 Scoring Guides/Criteria/ Rubric Data Table11

1a—Praxis II Content-Overall score data and subscore

data per standard

1b—PLT

NA

NA

Check the box if attached.

11

Licensure test data must reflect the percentage of candidates who have passed the state licensure test for each year

since the last accreditation visit. The most recent year of data must include the range of total scores and sub-scores

on the licensure test. Data must be presented for all program completers, even if there were fewer than 10 test takers

in a given year. Sub-score data tables will report the N, the % if candidates’ performance and the average

performance range provided in the Praxis report.

Assessment #2: (Required) PEDAGOGICAL AND PROFESSIONAL KNOWLEDGE AND SKILLS: Assessment that demonstrates

candidates can effectively plan classroom-based instruction. Examples of assessments include the evaluation of candidates’ abilities to develop

lesson or unit plans, individualized educational plans, needs assessments, or intervention plans. Provide assessment information as outlined in the

directions for Section IV. (For initial programs this will TWS Factors 1-4 and the data will be provided by The Teachers College.) Limit

of 2 pages excluding attachments.

Instructional Design Final Project (Instructional Design Documentation, a Sample Instructional Product, and an Evaluation of the

Sample Instructional Product)

Description:

All work in IT800 is used as formative assessments. The final project is the summative assessment. Approximately six weeks into the

course, students pick an instructional problem that can be addressed with an instructional program. Using the instructional design

process, students analyze the problem and proceed to design, develop, and evaluate an instructional program to solve the problem.

Each step concludes with the submission of a completed portion of the process as a formative assessment. Students then use all of the

information gathered throughout the semester to produce a final project that may serve as an Instructional Program Prototype, include

Instructional Design Documentation, a Sample Instructional Product, and an Evaluation of the Sample Instructional Product.

Throughout the semester, students apply the instructional design steps toward developing an instructional program or training module,

but they also analyze instructional design case studies. The Final Project is a summative culmination of the knowledge and skills

obtained during the course. The student demonstrates the ability to analyze and apply instructional design to an actual problem. In

12

developing a prototype, the students must also evaluate and decide upon an instructional technology to use to develop a sample

instructional product.

1. Students design and develop, and evaluate an Instructional Product that addresses an identified problem that can be addressed with

instruction (Standard 1).

2. The student applies the Instructional Systems Design (ISD) Instructional Design model to design, develop, and evaluation the

instructional product. Students analyze the intended audience of their instructional program and determine how to use instructional

technology to facilitate grouping students and addressing diverse student populations (Standard 4).

3. The Instructional Design Documentation demonstrates the students’ application of the Instructional Design process (Standard 7).

The three standards are all measured according to the final project rubric and have a number of points awarded. The scores are then

compared with the standard for the assignment. Students must score at the 80% level or above to have met the standard.

Alignment:

The Final Project demonstrates that the student has the ability, knowledge, and skills to develop, implement and reflect upon an

instructional problem.

Standard 1 - Evaluate and use computer and related technologies to support the instructional process.

As part of the final project, students design, develop and evaluate an Instructional Product that addresses an identified problem that can

be addressed with instruction. Students must evaluate instructional technologies and decide which technology to use to deliver

instruction. Students have used a variety of technologies, including word processing, web page design, and multimedia modules to

complete this portion of the final project. The specific criteria for the Final Project, or the Instructional Product, indicate whether the

student is able demonstrate how to evaluate and use computer and related technologies to support the instructional process. An

example is the specific instructional product that is created by the student.

Standard 4 - Design and develop student learning activities that integrate computing and technology for a variety of student grouping

strategies and for diverse student populations.

The design, development, and evaluation of the instructional product require students to analyze the intended audience, evaluate and

use instructional technologies to facilitate grouping strategies and addressing diverse student populations. The Instructional Design

Documentation demonstrates whether the student is able to know how to analyze an audience and develop appropriate strategies for

student grouping and addressing diverse student populations.

Standard 7 - Possess functional knowledge of currently accepted principles of design and implementation as they relate to careers in

the field of instructional design and technology

The Instructional Design Documentation is a typical instructional design approach in business/industry/education for sharing the design

process and findings of the Instructional Designer. The Client is the official representative from a business/industry/education

organization. This is often done in a short report (under ten pages) with supporting supplemental documents. The Instructional Design

Documentation indicates whether the student can provide a typical document that is used within the instructional design field to report

the findings of to a client.

Summary of Data:


35 of 40 students achieved the 80% competency required. N = 40, Mean = 88.8, Range 0-100

13





the field of instructional design and technology.


Interpretation of Data:

The data findings indicate that 35 of 40 students met Standards 1, 4, and 7 at the requisite 80% rate.

The course is designed so that most of the course work is formative and builds toward the Final Project (Instructional Design

Documentation, Sample Product). Students are given feedback during each formative assignment which is a component of the Final

Project. Upon beginning the Final Project, students have the ability, knowledge, and skills requisite for achieving the summative Final

Project. The departmental criterion of 80% for achieving standards was met by the majority of students as evidenced by the rubric.


Thirty-five of 40 students were able to use the computer and other related technologies to produce the required Sample Instructional

Product which displayed their solution to the instructional problem. Two students did not complete the requirements for the class and

did not officially withdraw from the class. The other three students experienced some difficulty with the analysis and design aspects of

the instructional design process.



Thirty-five of 40 students examined and analyzed the intended audience of the identified instructional problem evaluate and use

instructional technologies to facilitate grouping strategies and addressing diverse student populations. Two students did not complete

the requirements for the class and did not officially withdraw from the class. The other three students experienced some difficulty with

the analysis and design aspects of the instructional design process.



Thirty-five of 40 students were able to apply the Instructional Design steps of the Instructional Systems Design model to an

instructional problem and demonstrated the achievement of 80% competency in the Instructional Design Documentation. The other

three students experienced some difficulty with the analysis and design aspects of the instructional design process.

Attachments

Assessment #2 Scoring Guides/Criteria/ Rubric Data Table

[Assessment of candidate ability to plan instruction]

(Required); include name of the assessment:

Instructional Design Final Project

Check the box if attached X


X

Assessment #3 (Required) PEDAGOGICAL AND PROFESSIONAL KNOWLEDGE AND SKILLS: Assessment that demonstrates

candidates' knowledge and skills are applied effectively in practice. The assessment instrument used in student teaching should be submitted.

14

Provide assessment information as outlined in the directions for Section IV. (For initial programs student teacher data will be provided by

The Teachers College.) Limit of 2 pages excluding attachments.

MS Project Online Component

Description:

IT899 is the MS Project. It is the capstone course for the degree. Candidates must demonstrate knowledge and skills in a final project

that is oriented toward education, training, and/or careers. The final instructional design project focuses on a need of the student’s

school classroom, school building, school district, occupation, business or military unit. In their clinical setting, students determine an

instructional problem, determine what design and technology would solve the problem, develop the solution to the problem, implement

the solution (sometimes with the help of other stakeholders), and evaluate the success of their solution to the clinical problem. In a

presentation about their clinical setting, problem, and solution, students present a movie showing their work, a brochure outlining their

work, and a clinical report demonstrating the instructional design process used. Students are working in clinical settings. They

determine a problem, solve the problem using the ADDIE Model of Instructional Design, and present their solution to the problem.

Analysis is conducted by determining how the students implemented each part of the ADDIE Model – Analysis, Design, Development,

Implementation, and Evaluation. The success of each project is determined by how well the solution solves the problem. This is

measured by course rubrics.

Alignment with Standards:

Standard 2 - Apply current instructional principles, research, and appropriate assessment practices to the use of computers and related

technologies. The context for applying these components is the clinical setting in which the students work – education, military,

business, and/or industry and the specific occupational position that they hold. The students apply the assignments specifically to their

own clinical setting. This produces a structured framework of response but differing responses based on the students’ settings. See

Assignment Appendix files for specific assignment structures. Grades are not a part of the assessment of the standard.

Candidates must follow the ADDIE instructional design process during the implementation of their MS Project. ADDIE includes the

analysis, design, development, implementation and evaluation stages of instructional design and development. (Assignments #1, 2, 3,

4, 5, 6, 7, 8)

In Assignment #1 students determine an instructional design and technology problem that they could study within a clinical setting. In

Assignment #2 students provide a release form so that the Instructional Design & Technology faculty can share the students work

among themselves and other students. In Assignment #3 students determine whether or not they need a Human Subjects form for

working with people in their clinical setting. In Assignment #4 students do a needs analysis related to a specific instructional design

and technology problem. In Assignment #5 students design a solution for the perceived problem. In Assignment #6 students develop

their step-by-step process to determine a solution to their problem. In Assignment #7 students implement and carry out their solution to

the instructional problem. In Assignment #8 students evaluate (formative and summative evaluations), the success of the problem’s

solution within their clinical setting.

Standard 6 - Possess functional knowledge of practices, theories, major research findings and trends related to the field of

instructional design and technology.

Candidates must produce an instructional design digital movie and a client brochure. (Project Assignment #9, 10) In Assignment #9

students visually demonstrate the instructional design process that they used and provide their solution in a visual format. In

15

Assignment #10 students produce a brochure that provides their instructional design process, their problem, their solution, and their

clinical setting in order to provide a quick overview of their work.



Candidates must produce a client report and demonstrate that they can follow the instructional design process. (Assignments #11, 12)

In Assignment #11 students produce a comprehensive and specific report that demonstrates that they have used the instructional design

model, followed the ADDIE Model of instructional design, implemented any technological solutions, and determined a solution and its

successfulness. In Assignment #12, the Instructional Design & Technology faculty (advisor and committee members with input from

the remaining IDT faculty) evaluate the students use of an instructional design model, the following of the design model process, and

the carrying out of the process within the students’ clinical setting.

Summary of Data:

Standard 2: Of the 46 candidates, 42 met the 80% standard. N = 42, Range 490-500, Median 499



The data comes from the class enrollment during the fall 2009 and spring 2010. Meeting the standard is determined by the Rubric

Assessment.

Interpretation of Data: Forty-two (42) students achieved the minimum score of 80% for each of the standards. Four (4) students did not achieve the minimum

score and will be required to repeat the course. Students who received corrections by the IDT faculty and made revisions and

corrections met the minimum score. The students who received corrections but did not revise or correct their work did not meet the

minimum score and will retake the course with an opportunity to meet the standards.

Standard 2 - Apply current instructional principles, research, and appropriate assessment practices to the use of computers and related

technologies. Students worked within a clinical setting wherein they applied these components under the direction of IDT faculty.

Students were monitored and supervised during the process.


instructional design and technology. Students chose specific practices to implement which were based upon current research and trends

in instructional design and technology. The IDT faculty monitored, supervised, and suggested information to the students as they

applied their research.


the field of instructional design and technology. Students were monitored and supervised by IDT faculty so that corrections were

provided when errors occurred by the students. The students were provided the opportunity to revise their work during the clinical

setting so that their final presentation, paper, and process were appropriately completed by the end of the course.

Attachments


[Assessment of clinical experience*] (Required); Check the box if attached Check the box if attached.

16

Attachments


include name of the assessment:

MS Project Online Component

X X

*Clinical experience includes practica, student teaching and internships.

Assessment 4 (Required) EFFECTS ON STUDENT LEARNING: Assessment that demonstrates candidate effects on student learning.

Examples of assessments include those based on student work samples, portfolio tasks, case studies, follow-up studies, and employer surveys.

Provide assessment information as outlined in the directions for Section IV. (For initial programs this will the TWS Factors 5-7 and the data

will be provided by The Teachers College.) Limit of 2 pages excluding attachments.

MS Project Presentation/Paper

Description:

IT899 is the MS Project. It is the capstone course for the degree. Candidates must demonstrate knowledge and skills in a final project

that is oriented toward education, training, and/or careers. The final instructional design project focuses on a need of the student’s

school classroom, school building, school district, occupation, business or military unit. In their clinical setting, students determine an

instructional problem, determine what design and technology would solve the problem, develop the solution to the problem, implement

the solution (sometimes with the help of other stakeholders), and evaluate the success of their solution to the clinical problem. In a

presentation about their clinical setting, problem, and solution, students present a movie showing their work, a brochure outlining their

work, and a clinical report demonstrating the instructional design process used. See the Rubric Assessments Appendix that are used to

score the components of the students’ projects. Each rubric criteria is scored by the faculty teaching the course and/or the Instructional

Design and Technology Department faculty as a group.




Candidates must produce an instructional design Flash digital movie and a client brochure. (Project Assignment #9, 10) In Assignment

#9 students visually demonstrate the instructional design process that they used and provide their solution in a visual format. In

Assignment #10 students produce a brochure that provides their instructional design process, their problem, their solution, and their

clinical setting in order to provide a quick overview of their work. These assignments were aligned with the assessments and with the

standards so that upon completion of the assignments students would have met the standards as demonstrated in the assessments.

17



Candidates must produce a client report and demonstrate that they can follow the instructional design process. (Assignments #11, 12)

In Assignment #11 students produce a comprehensive and specific report that demonstrates that they have used the instructional design

model, followed the ADDIE Model of instructional design, implemented any technological solutions, and determined a solution and its

successfulness. In Assignment #12, the Instructional Design & Technology faculty (advisor and committee members with input from

the remaining IDT faculty) evaluate the students use of an instructional design model, the following of the design model process, and

the carrying out of the process within the students’ clinical setting. These assignments were aligned with the assessments and with the

standards so that upon completion of the assignments students would have met the standards as demonstrated in the assessments.

Summary of Data:




Forty-two (42) students achieved the minimum score for each of the standards. Four (4) students did not achieve the minimum score

and will be required to repeat the course. Students who received corrections by the IDT faculty and made revisions and corrections met

the minimum score. The students who received corrections but did not revise or correct their work did not meet the minimum score and

will need to retake the course with an opportunity to meet the standards.

Attachments


[Assessment of candidate effect on student learning*]

(Required); include name of the assessment:

MS Project Presentation/Paper



X

*Effects on student learning include the creation of environments that support student learning.

Assessment # 5: (Required) CONTENT KNOWLEDGE: Assessment of content knowledge. Examples of assessments include

comprehensive examinations, projects, comprehensive portfolio tasks and score/s aligned to standards OR the option of submitting course grades-

based assessment related to content knowledge evaluation. Provide assessment information as outlined in the directions for Section IV. If

submitting course grades-based assessment, the detailed description for Assessment #5 must clearly delineate the alignment of the course

description and assessments to the standard that is assessed during the course in order to assure that the course grade reflects candidate knowledge

of the standard. If course grades are used, include the program or unit definition of grades in the narrative form or as an attachment to Assessment

#5. If the course grades-based assessments are used as evidence for meeting two standards, the course key assessments’ data (exams, projects,

portfolio tasks) must be disaggregated in a data table for each of the two standards. This is necessary to provide evidence of meeting each standard.

This narrative must state the proficiency level or grade acceptable by the program. COURSE GRADES-BASED ASSESSMENTS ARE LIMITED

TO SIX COURSES. No more than 5 pages.

18

Multimedia Project Demonstrate Quality Multimedia Development by Completing the Cycle of Designing, Developing, Critiquing and Elaborating on an

Individual Flash Project. Description: Several projects in IT 810 lead students through the process of effectively developing multimedia projects. The ENTIRE RUBRIC for

the final Flash project showcases students ability to effectively use multimedia in training or education as articulated by Standards 3

and 5. More specifically, this project demonstrates the student’s ability to utilize the instructional design process to develop

instructional multimedia. The final Flash project gives students a forum to both showcase their ability to develop effective educational

multimedia and to reflect on what they would change if they were to engage in the process again after reviewing and reflecting on the

work of their peers and their own process of development.

Factor 1 - Storyboard a multimedia project for development in Flash (Standard 3)

Storyboard development demonstrates an ability to visualize the use and integration of any number of the building blocks created in

factor 1.

Factor 2 - Create specific content elements in a number of software programs for integration into a Flash project.(Standard 5)

Demonstrating an ability to create these elements showcases a fundamental ability to develop the building blocks of multimedia

instruction. Factor 1 is worth 20% of the standard score (each element is worth one percentage point).

Factor 3 - Develop a multimedia project in Flash that demonstrates effective instructional design as measured by the quality of both

the instructional content and design elements (Standard 5)

Factor 4 - Review and reflect on peer projects. Discuss version 2 of your project. (Standard 3)

This development process articulates how the development of the project can be improved. Alignment with Standards:

Standard 3 - Demonstrate knowledge of computer usage for problem solving, data collection, information management, communications,

presentations, and decision-making utilizing multimedia, hypermedia and telecommunications technologies.

This is showcased in factors 1 and 4. In factor 1, students storyboard a multimedia Flash project which plans for proper integration of

animations, various media, effective navigation, and/or linking elements. In factor 4, students complete the cycle of effective

instructional design by reviewing other students projects, reflecting on elements that would enhance their learning project, and then

participating in a discussion by articulating how they would improve their project in a second round of development. Standard 5- Demonstrate knowledge of multimedia, hypermedia, distance education and online learning theories and practices to support

instruction. In factors 2 and 3, students demonstrate that they can create specific content elements in several different software programs. A

number of visual and audio software programs can be used to develop media elements that can be integrated into Flash. These elements are

continually changing as software evolves requiring updates every semester.

Summary of Data:

Standard 3 - Data findings showcase that 47 students met this standard while 5 students failed to meet this standard, scoring less than

6 points of the 8 points available for factors one and four. The mean score for this standard was 7.4, with a range from 5 to 8 points.

This data does not include three students that did not complete the course and did not receive points for these activities.

Standard 5 - Data findings showcase that 47 students met this standard while 5 students failed to meet this standard by scoring less

than 13 points of the 18 points available for factors two and three. The mean score for this standard was 15.1, with a range from 12 to

17 points. This data does not include three students that did not complete the course and did not receive points for these activities.

19

Failure to meet the standards did have a major impact on student grades as each point taken off for missed standards equates to a loss

of one percentage point. The 25 possible points for this final project, equates to 25 percent of the student’s grade in the course. Some

students barely passed this graduate level course with C grades. Interpretation of Data:

Standard 3 - Storyboards development was strong for most students. An online tool makes storyboarding both easy to do and a highly

effective tool in project development. Peer reviews of projects were very helpful for continued development. Final self-evaluations of

personal projects were not up to standards. After further analysis, it appears that most students were ineffective because they chose not

to participate in this assignment likely because they had already earned an A in the class, which effectively meant that this assignment

would not help their grade.

Standard 5 - Students struggled with three items in this Standard. First, students had trouble integrating multiple media elements into

one project (rubric item 2.1). Second, students had difficulty in effectively making these media elements work well together (rubric

item 2.3.1). Finally, students had the most trouble in effectively using the more sophisticated tools available in Flash (rubric item 3.4).

These are challenging skills that come with practice. However, after much reflection and discussion as a department, it has been

decided that the course focus of both learning to use a difficult program (Flash) AND learning how to develop effective media may be

too much for most students to handle. Can they do it? Yes they can. But the deeper, more probing question relates to how much are

they retaining and able to effectively use once the course is over. Only a few will use Flash once they leave ESU but all need to learn to

develop effective media. We are now exploring how to move the course to focus more on development. The rubric would not change

much. Students would still need to work on the three trouble areas listed above. With the new focus, students would pick a tool they are

more likely to use in the future. The focus would then be on effectively integrating media elements using that tool.

The majority of students achieved the standards. However, a tenth of our students did not. There are several reasons for this.

1. We are shooting at a moving target. The standards for this course have been built on a foundation of software development.

Software is continually evolving. A new focus will be to target effective media integration using Richard A. Mayer’s 12 rules

of multimedia learning as targeted in the second edition of his book titled Multimedia Learning. These rules will not change as

much as software changes, so the foundation of the course should become a bit more solid.

2. By far the most difficult part of this standard is rubric item 3.4 (Sophisticated use of Flash using ActionScripts and transitions).

The reason for this is time. Anyone can learn to use the more difficult tools in Flash if they have time. However, learning this

AND learning how to effectively integrate media at the same time is just too much. Additionally, if most students never plan on

using Flash again, this is a skill that will go nowhere. After much departmental reflection, it was decided that students can

always take an online course or get a book on Flash and learn it on their own. What is more difficult to learn on their own is

how to effectively develop and integrate media in developing a media project. That is what we plan to focus on.

20

Attachments


[Content based assessment that addresses Kansas

content standards] Required Examples of

assessments include comprehensive examinations,

projects, comprehensive portfolio tasks and score/s

aligned to standards; include name of the assessment:

Multimedia Project



X

IF COURSE GRADES-BASED ASSESSMENTS are submitted, the following matrix MUST be used in addition to the

narrative detailed description of the assessments the program provides in the above #5 Content Knowledge description!

Alignment Matrix and Course Description for Course Grades-Based Assessment

Assessments 5.A-

F for SIX courses

Course Name &

Number

Program Standard

Addressed by Course

Assessment

Brief Description of how the Course addresses and assesses the standard from an

AUTHENTIC source—such as a syllabus or course catalog. Cite the most current source.

The description should provide evidence of the alignment of the course to the standard

indicated on the chart in Section III. Cite your source in each description below.

EXAMPLE:

Calculus I

Math 172

Standard 6 Calculus of algebraic functions of one variable: limits differentiation, implicit differentiation,

definite and indefinite integrals. Mean value theorem, maxima and minima, area, and volume.

Vectors, polar coordinates, parametric equations, and vector valued functions and use of

technology. Applications to other fields.

Source: Blank University Undergraduate Catalog

5.A.

5.B.

5.C.

5.D.

5.E.

5.F.

Assessment 6: (Required) CONTENT KNOWLEDGE: Assessment of content knowledge. Examples of assessments include comprehensive

standard examinations, case studies involving many content standards, projects, comprehensive portfolio tasks and score/s aligned to standards and

follow-up studies related to content knowledge. Provide assessment information as outlined in the directions for Section IV. Limit of 2 pages

excluding attachments.

21

Web-Based Design Project

(Demonstrate design and content competencies by completing the final project of designing a teaching unit for web-based training)

Description:

Design and create a unit for web-based training based on the strategies learned throughout the semester and in other IT classes.

Students are encouraged to develop their projects to use in their work so as to enhance relevancy to real world situations. Projects are

assessed on content, design, multimedia, and implementation of knowledge of web-based training practices.

Standard 3 - Content: Based on the course textbook readings, online presentations, and student discussions of Instructional Design and

Technology theory related to Web-Based Training, learners will apply concepts to the generation of their course project. Completion of

this project requires students to conduct research, collect data, problem solve by applying the project content to their specific discipline

area, generating related multimedia and communications media, creating and organizing new content generated, editing and revising,

then presenting the final product to their classmates and instructor.

Standard 4 - Design: The project requirements include the generation of technology-based student learning activities. The project is

designed for diverse student populations using a variety of media.

Standard 5 - Multimedia: Students will demonstrate online learning theories and practices to support quality instructional design, as it

may be used in their specific professional settings. Their individual Web-Based Training sites will be created using online multimedia

authoring tools.


Standard 3 - Content: Demonstrate knowledge of computer usage for problem solving, data collection, information management,

communications, presentations, and decision-making utilizing multimedia, hypermedia and telecommunications technologies.

The final project for this class incorporates the processes included in Standard 3 through the production of a multi-media web-based

training unit reflecting the students’ knowledge of and ability to implement essential components of an effective teaching unit that can

be used for online learning. Appropriate content was generated through research, data collection, problem solving, information

management, communications, presentations, and decision-making. All facets were used to create the Web-Based Training course

project. The development process included content overviews’, objectives, standards, lessons, topic and audience, policies, syllabus,

grading, and a project based learning activity.

Standard 4 - Design: Design and develop student learning activities that integrate computing and technology for a variety of student

grouping strategies and for diverse student populations. Students demonstrated their knowledge of effective technology design in their

website projects by including clear layouts, easy navigation of links, effective use of color and fonts, and correct mechanics of spelling

and grammar in order to effectively reach diverse learning populations.

Standard 5 - Multimedia: Demonstrate knowledge of multimedia, hypermedia, distance education and online learning theories and

practices to support instruction. Students demonstrated their knowledge of online learning theories through the effective creation of a

multimedia based learning environment for Web-Based Training. Students generated online instruction for their targeted audience

reflecting their understanding of multimedia technologies to support electronic collaboration, exemplars, media, feedback, technology

requirements, pre and post assessments, supplemental resources, instructor autobiography with contact information, course resource

materials, and a calendar, checklist or schedule to support instruction.

Summary of Data:

22

Standard 3 - Of the 48 candidates, 41 met the standard and 7 did not reach the minimum standard. N = 48, Mean = 88%, Range 0-10



Interpretation of Data: Standard 3 - Demonstrate knowledge of computer usage for problem solving, data collection, information management,

communications, presentations, and decision-making utilizing multimedia, hypermedia and telecommunications technologies.

In this assessment, all students participating in and completing the Web-Based Training project were successful at meeting standard 3

with 8 exceptions. Three students not completing the project and five students lacking content quality did not meet the standard. Based

on the statements provided by those students not completing the coursework, the reasons cited were an accumulation of personal issues

due to personal or family illnesses, overbooking responsibilities, and work conflicts. We will work towards nurturing students with

personal issues to encourage them to complete the course. We will offer project checkpoints before the due date to assist in monitoring

content quality success.


strategies and for diverse student populations. In this assessment, all students participating in and completing the Web-Based Training

project were successful at meeting standard 4 with four exceptions. Three students not completing the project and one students lacking

content quality did not meet the standard. Based on the statements provided by the student not completing the coursework, the reasons

cited were an accumulation of personal issues due to personal or family illnesses, overbooking responsibilities, and work conflicts. We

will work towards nurturing students with personal issues to encourage them to complete the course.

Standard 5 - Demonstrate knowledge of multimedia, hypermedia, distance education and online learning theories and practices to

support instruction. In this assessment, all students participating in and completing the Web-Based Training project were successful at

meeting standard 5. Three students not completing the project did not meet the standard and two students lacking content quality did

not meet the standard. Based on the statements provided by those students not completing the coursework, the reasons cited were an

accumulation of personal issues due to personal or family illnesses, overbooking responsibilities, and work conflicts. We will work

towards nurturing students with personal issues to encourage them to complete the course. In addition, for students having difficulty

managing their class time and assignments, there is now a clear set of expectations set forth on a weekly basis for the students to use to

be successful in this class.

The course is designed so that much of the work is formative and builds toward the Final Web-based Training Project. Students are

given feedback during the steps of the final project. Standard 5 reflects an overall efficient and well-designed product while Standards

3 and 4 are specific factors (content and design) to be considered. Meeting Standards 3 and 4 does not necessarily mean that Standard 5

will be met even though Standards 3 and 4 could be construed as components of Standard 5

Attachments


[Content based assessment that addresses Kansas

content standards] Required Examples of



X

23

Attachments


assessments include comprehensive standard

examinations, case studies involving many content

standards, projects, comprehensive portfolio tasks

and score/s aligned to standards, and related to

content knowledge; include name of the assessment:

Web-Based Design Project

Assessment 7: (Optional) Additional assessment that addresses Kansas content standards. Examples of assessments include

evaluations of field experiences, case studies, portfolio tasks, licensure tests not reported in #1, and follow-up studies. Provide

assessment information as outlined in the directions for Section IV. Limit of 2 pages excluding attachments

Issues in Distance Education Final Project (Literature Review and Multimedia Presentation)

Description:

All work in IT830 is used as formative assessments. The final project is the summative assessment. Throughout the semester, students

are introduced to different issues related to distance education. The Final Project is a summative culmination of the knowledge and

concepts obtained during the course. After a survey of the varying issues in distance education, the student demonstrates the ability to

conduct research on a specific distance education issue through a literature review. The student then develops an instructional product

by developing a multimedia presentation/module about the research topic.

1. Students review distance and online technologies throughout the course. In the final project, students evaluate and select one of

these technologies to design and develop an instructional multimedia presentation or module about a specific issue in distance

education (Standard 1).

2. The student critically analyzes the literature of a specific distance education topic and develops an instructional multimedia

presentation (Standard 5).

3. The literature review and multimedia presentation demonstrates the students’ functional knowledge of practices, theories, major

research findings and trends related to the field of instructional design and technology. (Standard 6).

The Contemporary Issues in Distance Education Final Project is scored according to the following rubric. The three standards are all

measured according to the final project rubric and have a number of points awarded. The scores are then compared with the standard

for the assignment. Students must score at the 80% level or above to have met the standard.

Alignment: The Final Project demonstrates that the student has the knowledge and skills necessary to research and critically analyze issues in

distance education. In addition, the student demonstrates his/her ability to design and develop an instructional multimedia product.


As part of the final project, students design and develop an instructional multimedia presentation or module about a specific issue in

distance education. Throughout the course, students are introduced to different distance and online technologies. Students must

evaluate these technologies and decide which technology is best to deliver instruction.

24


support instruction.After reviewing topics related to multimedia, hypermedia, distance education, and online learning theories and

practices to support instruction, students select one issue already discussed in the course to conduct an in-depth research literature

review. Within the literature review, students must critically analyze the issue in relation to distance education and online learning

theories and practices.


instructional design and technology. Issues in Distance Education is a major topic under the overarching field of instructional design

and technology. Through the completion of a literature review and instructional multimedia presentation, the student demonstrates

his/her knowledge of practices, theories, major research findings and trends related to the field of instructional design and technology.

Summary of Data:


46 of 50 students achieved the 80% competency required. N = 50, Mean = 91, Range 0-100


support instruction






The data findings indicate that 46 of 50 students met Standards 1, 5, and 6 at the requisite 80% rate.

The majority of students met the departmental criterion of 80% for achieving standards as evidenced by the final project rubric.


Forty-six of 50 students were able to use the computer and other related technologies to design and develop an instructional multimedia

presentation. One student did not complete the requirements for the class and did not officially withdraw from the class. The other

three students while, completing the requirements, did not have a high quality product.


support instruction Forty-six of 50 students demonstrated knowledge of multimedia, hypermedia, distance education and online

learning theories and practices to support instruction through the completion of a literature review and multimedia presentation. One

student did not complete the requirements for the class and did not officially withdraw from the class. The other three students while,

completing the requirements, did not have a high quality product.


instructional design and technology. Forty-six of 50 students were able to demonstrate their knowledge of practices, theories, major

research findings and trends related to the field of instructional design and technology through the completion of a literature review and

instructional multimedia presentation. One student did not complete the requirements for the class and did not officially withdraw

from the class. The other three students while, completing the requirements, did not have a high quality product.

25

Attachments


[Optional. Additional assessment that addresses

Kansas content standards ]; include name of the

assessment:

Issues in Distance Education Final Project



X

Assessment 8: (Optional) Additional assessment that addresses Kansas content standards. Examples of assessments include evaluations of

field experiences, case studies, portfolio tasks, licensure tests not reported in #1, and follow-up studies. Provide assessment information as outlined

in the directions for Section IV. Limit of 2 pages excluding attachments

Attachments


[Optional. Additional assessment that addresses

Kansas content standards ]; include name of the

assessment:

Check the box if attached


26

SECTION V—USE OF ASSESSMENT RESULTS TO IMPROVE

CANDIDATE AND PROGRAM PERFORMANCE Evidence must be presented in this section that assessment results have been analyzed and have been or will be used to improve candidate

performance and strengthen the program. This description should not link improvements to individual assessments, but rather, it should summarize

major findings from the evidence, the faculty’s interpretation of those findings, and changes made in (or planned for) the program as a result.

Describe the steps program faculty have taken to use information from assessments for improvement of both candidate performance and the

program. Limit of three pages. (This will be a summary of the Data Results and Action Taken/Recommendations from the

Assessments and Related Data chart in Section II.)

Based on the candidate performance on assessments, several curriculum changes (within courses) are anticipated:

1) Because of ever-changing software version changes, the anchor applications in IT810 Multimedia Design will be changed

and/or updated to include appropriate multimedia authoring software.

2) Due to difficulties with mastering Flash in IT810, the faculty decided to have candidates concentrate on how to effectively

develop and integrate media in developing a media project.

3) Textbook and software requirements for candidates taking IT810 will be updated to match new course outcomes and materials.

4) All courses will have a greater emphasis on writing and appropriate APA style.

5) Due to some difficulty in locating and retrieving resources for IT 800 and IT 899, more emphasis will be given to locating and

retrieving online research articles through the ESU library.

6) IT 820 Designing/Developing Web-Based Instruction will be updated to take into consideration the latest advances in web-

based and e-learning tools.

7) Because of the need for stronger backgrounds in teaching and learning and because there is an increased number of candidates

with non-education backgrounds in the program, PY805/PY811 were added as core courses for the program.

8) Instructional Design principles from IT800 Instructional Design were fully integrated into the master’s project and paper.

9) The structure of IT 899 Master’s Project was changed to enable candidates to spend more time and resources on the actual

project itself, instead of writing a paper.

10) To emphasize multimedia design and development skills, a Flash movie presentation and brochure replaced the original

PowerPoint presentation in the Master’s Project.

11) To ease uploading and prompt feedback, IT 899 Master’s Project videos were adapted to YouTube format and uploaded to

YouTube.

12) LI 813 was dropped as a core requirement from the program and outcomes were addressed in IT 710.

13) Due to some difficulties by candidates and analysis and design, more attention will be given to the analysis and design stages

of instructional design in IT 800.

14) Based upon trends, it appears that an increasing number of candidates are encountering problems completing their master’s

project during their intended semesters (sometimes due to physical/emotional or family issues of which faculty have no

27

control). However, more attention will be given to advising, so that candidates can make smart enrollments decisions (number

of credits per semester, semester load, taking work and family into consideration).

15) All course program changes (such as new courses and titles) must be approved by the IDT faculty, the Committee on Advanced

Programs, the Academic Leadership Council, and are then forwarded for university approval.

VI—RESPONSE TO AREAS FOR IMPROVEMENT FROM LAST KSDE REVIEW

(List areas for improvement cited and what has been done to correct each.)

28

H/TC/Common/AnnuaProgramRpts07-08/ProgAnnualRptBlk07-08

esu revised version 6/15/10 instructional … · 3 section i —context complete ... integrating...

Documents