assessment review panel responses to dispositions rubric 2017 … · teacher education assessment...

Assessment Review Panel Responses

to

Dispositions Rubric

2017

1

Teacher Education Assessment Review Panel

Dispositions Rubric

2017

The Teacher Education Assessment Review Panel was created in June 2017. Its charge is to review and provide

critical feedback and analysis to the Purdue University Teacher Education Program faculty and staff about its

assessment instruments. This panel is part of the continuous improvement efforts of Teacher Education at Purdue.

Membership consists of five to six (5-6) individuals, who are familiar with assessment or experts in their field,

and selected by the Associate Dean for Learning. Ideally, these individuals are external to Purdue University and

consist of two (2) members from P-12 partner schools, two (2) members from higher education, and one (1) other

member from P-12, higher education, or another agency/organization.

The assessment instrument and any other materials are provided to the panel members and distributed for review

and feedback through email messaging or a SharePoint site. The Council for the Accreditation of Educator

Preparation (CAEP) Evaluation Framework for EPP-Created Assessments (see Appendix A) is used in this

process.

The dispositions document (see Appendix B) was shared with five (5) panel members via email message with a

link to Qualtrics, a web-based survey creations, collection, and analysis software tool. Four (4) panel members

responded to the survey. Following are their responses.

ADMINISTRATION AND PURPOSE (informs relevancy)

# Answer Response %

1 Below Sufficient Level 0 0.00%

2 Sufficient Level 1 25.00%

3 Above Sufficient Level 3 75.00%

Total 4 100.00%

Min

Value Max Value Variance

Standard

Deviation

Total

Responses

2 3 0.25 0.5 4

CONTENT OF ASSESSMENT (informs relevancy)

# Answer Response %




Total 4 100.00%

http://caepnet.org/~/media/Files/caep/accreditation-resources/caep-assessment-tool-v1-20170127t140453.pdf?la=en

http://caepnet.org/~/media/Files/caep/accreditation-resources/caep-assessment-tool-v1-20170127t140453.pdf?la=en

2

Min


Standard

Deviation

Total

Responses

2 3 0.333333333 0.577350269 4

SCORING (informs reliability and actionability)

# Answer Response %




Total 4 100.00%

Min


Standard

Deviation

Total

Responses

2 3 0.333333333 0.577350269 4

DATA RELIABILITY

# Answer Response %




Total 4 100.00%

Min


Standard

Deviation

Total

Responses

1 3 1 1 4

DATA VALIDITY

# Answer Response %




Total 4 100.00%

Min


Standard

Deviation

Total

Responses

1 3 1 1 4

3

OVERALL RATING AND COMMENTS

Text Entry

Acceptable

I understand the "not applicable/observable" category, but it seems to me that in most, if not all, of these

categories, the scorer should have the opportunity to observe these behaviors.

Will this tool/rubric be used by cooperating teachers?

There are multiple uses of the word ‘behavior(s)” including #4 titled Professional Behavior. Might there

be a way to either very clearly define the meaning for each or use a different term instead for specific

circumstances? (Some I looked up might be deportment, bearing, presence, actions, but I am sure there

are others).

In the second paragraph of the introduction of the rubric there is a reference to professional dispositions.

Where and how are candidates and scorers made aware as to the meaning of this?

In the third paragraph there is mention of a professionalism agreement. By any chance does this include

an ethics piece?

Could a link to the NASDTEC Model Code of Ethics be included as a resource? Or one where PU has

adapted the MCEE to fit their specific needs?

I think it is important that candidates and scorers understand that conduct and ethics are not the same

thing. For #1, how is ethical conduct defined? Is it specifically only those listed in the description of #1?

Is there a definition of academic integrity anywhere? That is mentioned in #1.

Should all of the numbered items begin with “Commitment to ----“? Or is this intentional for only

specific ones (#2, 3, 5, 8)

Thanks for giving me the opportunity to give input. Looking forward to our working together again this

year!

Email Response

You have created a quality instrument. The issues with the Data Reliability and Validity sections can be

addressed in a separate document and should satisfy a CAEP team. You might also consider using the

dispositions assessment document as a survey as this instrument does address those criteria at the

sufficient level.

Four suggestions:

1. Mention InTASC alignment in the introduction and consider tagging each of the criteria to

InTASC.

2. Paragraph 4 of the introduction refers to foundations courses. Consider clarifying whether these

are in the Foundational Core or Professional courses.

3. I like the way you handled the Legal and Ethical Conduct section – only 2 options. You may

want to add a reference to social media for this criterion, too.

4. In the Commitment to the Profession section, consider using all caps in ALL students in the first

line, the same as in the third line.

4

Appendix A

CAEP EVALUATION FRAMEWORK

FOR EPP-CREATED ASSESSMENTS

For use with: Educator preparation provider (EPP)-created assessments, including subject and

pedagogical content tests, observations, projects, assignments, and surveys

For use by: EPPs to evaluate their own assessments and by CAEP site teams to review evidence in

self-study submissions

CAEP uses the term “assessments” to cover content tests, observations, projects or assignments, and surveys. All of these assessment forms are used with candidates. Surveys are often used to gather evidence on aspects of candidate preparation and candidate perceptions about their own readiness to teach. Surveys are also useful to measure the satisfaction of graduates or employers with preparation and the perceptions of clinical faculty about the readiness of EPP completers.

Assessments and scoring guides are used by faculty to evaluate candidates and provide them with feedback on their performance. Assessments and scoring guides should address relevant and meaningful attributes of

candidate knowledge, performance, and dispositions, aligned with standards. Most assessments that comprise evidence offered in accreditation self-study reports will probably be used by an EPP to examine candidates consistently at various points from admission through exit. These are assessments that all candidates are expected to complete as they pass from one stage of preparation to the next, or that are used to monitor progress of candidates’ developing proficiencies during one or more stages of preparation.

CAEP site teams will follow the guidelines in this evaluation tool and it can also be used by EPPs when they design, pilot, and judge the adequacy of the assessments they create.

5

EXAMPLES OF ATTRIBUTES BELOW SUFFICIENT LEVEL CAEP SUFFICIENT LEVEL

EXAMPLES OF ATTRIBUTES ABOVE SUFFICIENT LEVEL

- a. Use or purpose are

ambiguous or vague. b. There is limited or no

basis for reviewers to know what information is given to candidates.

c. Instructions given to candidates are incomplete or misleading.

d. The criterion for success is not provided or is not clear.

1. ADMINISTRATION AND PURPOSE (informs relevancy)

a. The point or points when the assessment is administered

during the preparation program are explicit. b. The purpose of the assessment and its use in candidate

monitoring or decisions on progression are specified and appropriate.

c. Instructions provided to candidates (or respondents to surveys) about what they are expected to do are informative and unambiguous.

d. The basis for judgment (criterion for success, or what is “good enough”) is made explicit for candidates (or respondents to surveys).

e. Evaluation categories or assessment tasks are aligned with CAEP, InTASC, national/professional and state standards.

+ a. The purpose of the

assessment and its use in candidate monitoring or decisions are consequential.

b. Candidate progression is monitored and information is used for mentoring.

c. Candidates are informed how the instrument results are used in reaching conclusions about their status and/or

progression.

-

2. CONTENT OF ASSESSMENT (informs relevancy)

a. Indicators assess explicitly identified aspects of CAEP, InTASC, national/professional and state standards.

+

6



a. Indicator alignment with CAEP, InTASC, national/ professional or state standards is incomplete, absent or only vaguely related to the content of standards being evaluated.

b. Indicators fail to reflect the degree of difficulty described in the standard.

c. Indicators not described, are ambiguous, or include only headings.

d. Higher level functioning, as represented in the standards, is not apparent in the indicators.

e. Many indicators (more

than 20% of the total score) require judgment of candidate proficiencies that are of limited importance in CAEP, InTASC, national/professional, and/or state standards.

b. Indicators reflect the degree of difficulty or level of effort described in the standards.

c. Indicators unambiguously describe the proficiencies to be evaluated.

d. When the standards being informed address higher level functioning, the indicators require higher levels of intellectual behavior (e.g., create, evaluate, analyze, & apply). For example, when a standard specifies that candidates’ students “demonstrate” problem solving, then the indicator is specific to candidates’ application of knowledge to solve problems.

e. Most indicators (at least those comprising 80% of the total score) require observers to judge consequential attributes of candidate proficiencies in the standards.

[NOTE: the word “indicators” is used as a generic term for assessment items. For content tests, the term refers to a question. For projects or assignments, it refers to a prompt or task that the candidate is to perform. For an observation, an indicator might be a category of performance to observe or a specific aspect of candidate performance that a reviewer would record. For a survey, an indicator would stand for a question or statement for which a response is to be selected.]

a. Almost all indicators (95% or more of the total score) require observers to judge consequential attributes of candidate proficiencies in the standards.

- a. Rating scales are used

instead of rubrics; e.g., “level 1= significantly below expectation” “level 4 = significantly above expectation.”

b. Proficiency Level Descriptors (PLDs) do not align with indicators.

c. PLDs do not represent developmental progressions.

d. PLDs provide limited or no feedback to candidates specific to their performance.

e. Proficiency level descriptors are vague or not defined, and may just

3. SCORING (informs reliability and actionability)

a. The basis for judging candidate performance is well defined.

b. Each Proficiency Level Descriptor (PLD) is qualitatively defined by specific criteria aligned with indicators.

c. PLDs represent a developmental sequence from level to level (to provide raters with explicit guidelines for evaluating candidate performance and for providing candidates with explicit feedback on their performance).

d. Feedback provided to candidates is actionable—it is directly related to the preparation program and can be used for program improvement as well as for feedback to the candidate.

e. Proficiency level attributes are defined in actionable, performance-based, or observable behavior terms. [NOTE: If a less actionable term is used such as “engaged,” criteria are provided to define the use of the term in the context of the category or indicator.]

+ a. Higher level actions from

Bloom’s or other, taxonomies are used in PLDs such as “analyzes” or “evaluates.”

7

Version 1 | January 2017 | 2



repeat the language from the standards.

- a. Description of or plan to

establish reliability does not inform reviewers about how it was established or is being investigated.

b. Described steps do not meet accepted research standards for reliability.

c. No evidence, or limited evidence, is provided that scorers are trained, and their inter-rater agreement is documented.

d. Described steps do not meet accepted research standards for reliability.

4. DATA RELIABILITY

a. A description or plan is provided that details the type of reliability that is being investigated or has been established (e.g., test-retest, parallel forms, inter-rater, internal. consistency, etc.) and the steps the EPP took to ensure the reliability of the data from the assessment.

b. Training of scorers and checking on inter-rater agreement and reliability are documented.

c. The described steps meet accepted research standards for establishing reliability.

+ a. Raters are initially,

formally calibrated to master criteria and are periodically formally checked to maintain calibration at levels meeting accepted research standards.

b. A reliability coefficient is reported.

8

- a. Description of or plan to

establish validity does not inform reviewers about how it was established or is being investigated.

b. The type of validity established or investigated is miss- identified or not described.

c. The instrument was not piloted before administration.

d. Process or plans for data analysis and interpretation are not presented or are superficial.

e. Described steps do not meet accepted research standards for establishing validity. For example, validity is determined through an internal

review by only one or two stakeholders.

5. DATA VALIDITY

a. A description or plan is provided that details steps the EPP has taken or is taking to ensure the validity of the assessment and its use.

b. The plan details the types of validity that are under investigation or have been established (e.g., construct, content, concurrent, predictive, etc.) and how they were established.

c. If the assessment is new or revised, a pilot was conducted. d. The EPP details its current process or plans for analyzing

and interpreting results from the assessment.

e. The described steps meet accepted research standards for establishing the validity of data from an assessment.

+ a. Types of validity

investigated go beyond content validity and move toward predictive validity.

b. A validity coefficient is reported.

Version 1 | January 2017 | 3

9

Criteria listed below are evaluated during the stages of the accreditation review and decisionmaking:

EPP provides evidence that assessment data are compiled and tabulated accurately

Interpretations of assessment results are appropriate for the items and resulting data

Results from successive administrations are compared (for evidence of reliability) Version 1 | January 2017 | 4

EXAMPLES OF ATTRIBUTES

BELOW SUFFICIENT LEVEL CAEP SUFFICIENT LEVEL EXAMPLES OF ATTRIBUTES

ABOVE SUFFICIENT LEVEL review by only one or two stakeholders.

WHEN THE INSTRUMENT IS A SURVEY: Use Sections 1 and 2, above, as worded and substitute sections 6 and 7, below for sections 3, 4 and 5.

- a. Questions or topics are

not aligned with EPP mission or standards.

b. Individual items are ambiguous or include more than one subject.

c. There are numerous leading questions.

d. Items are stated as opinions rather than as behaviors or practices.

e. Dispositions surveys

provide no evidence of a

relationship to effective

teaching.

6. SURVEY CONTENT

a. Questions or topics are explicitly aligned with aspects of the EPP’s mission and also CAEP, InTASC, national/professional, and state standards.

b. Individual items have a single subject; language is unambiguous.

c. Leading questions are avoided. d. Items are stated in terms of behaviors or practices instead

of opinions, whenever possible. e. Surveys of dispositions make clear to candidates how the

survey is related to effective teaching.

+ a. Scoring is anchored in

performance or behavior demonstrably related to teaching practice.

b. Dispositions surveys make an explicit connection to effective teaching.

- a. Scaled choices are

numbers only, without qualitative descriptions linked with the item under investigation

b. Limited or no feedback provided to the EPP for improvement purposes

No evidence that

questions/items have been

piloted

7. SURVEY DATA QUALITY

a. Scaled choices are qualitatively defined using specific criteria aligned with key attributes.

b. Feedback provided to the EPP is actionable. c. EPP provides evidence that questions are piloted to

determine that candidates interpret them as intended and modifications are made if called for.

+ a. EPP provides evidence of

survey construct validity derived from its own or accessed research studies.

10

Appendix B

Teacher Education Council

Candidate Disposition Assessment Rubric

The Teacher Education Council (TEC) is responsible for the oversight of teacher education programs at Purdue that

prepare candidates who have the required knowledge, skills, and dispositions to become effective teachers. Faculty,

supervisors, and school personnel will evaluate candidates at various points in the program and provide candidates with

feedback about progress.

The Council for the Accreditation of Educator Preparation (CAEP) requires that all candidates demonstrate the

professional dispositions associated with effective educators (Standard 3.3). These dispositions are defined as:

“the values, commitments, and professional ethics that influence behaviors towards students, families, colleagues,

and communities that affect student learning, motivation, and development as well as the educator’s own

professional growth.” (CAEP, 2016, http://caepnet.org/glossary).

Each candidate in Purdue University’s teacher education program is required to read and sign a Professionalism

Agreement (add link here) prior to participating in any field experience. This Professionalism Agreement (PA) outlines

the Teacher Education Council’s (TEC) expectation for candidates participating in field experiences in P-12 schools.

The rubric that follows mirrors the PA and serves as a tool for documenting the degree to which candidates demonstrate

these important attributes across all coursework and field experiences. Candidate dispositions will be assessed in each

foundations course and in all professional education (i.e., methods courses) courses as well as in student teaching.

Candidates will complete a self-assessment, and each course instructor/supervisor will complete an assessment. The

course instructor/supervisor will also review the candidate’s assessment, and the candidate and instructor/supervisor may

discuss observations of behavior in coursework and field or clinical experiences. Instructors/supervisors may document

‘areas for improvement’ in the candidate’s assessment. It is the candidate’s responsibility to address these areas. If

multiple ratings of ‘unacceptable’ occur, or candidates fail to address areas for improvement over multiple semesters, a

Concern Notice will be filed, a formal meeting to discuss recurring dispositional issues may be convened, and a D-2 (add

link here) may be filed. (NOTE: If candidates fail to meet Target expectations for ‘1.Legal and Ethical Conduct,” a D-2

may be filed regardless of the results of ratings of candidate performance on the remaining dispositions.)

1. Legal and Ethical Conduct: Candidate maintains professional, legal, and ethical conduct at all times (for example, reporting cases of abuse and neglect). Candidate is aware of FERPA and respects the privacy of children, families, and school personnel and protects the confidentiality of academic or personal information that he/she encounters. Candidate demonstrates academic integrity and does not engage in academic dishonesty (e.g., plagiarism).

Unacceptable Target Not Applicable/ Observable

Areas for Improvement:

Candidate fails to exhibit--or exhibits on an inconsistent basis—behaviors associated with legal and ethical conduct.

Candidate consistently and appropriately exhibits behaviors associated with legal and ethical conduct.

No opportunity to observe candidate exhibiting behaviors associated with legal and ethical conduct.

http://caepnet.org/glossary

11

2. Commitment to the Profession: Candidate demonstrates a commitment to all students and their learning by being fully prepared for all activities, including instructional planning. Candidate positively encourages students in their learning efforts and maintains high expectations for ALL students. Candidate demonstrates a commitment to his/her field of study, students, and to the teaching profession.

Unacceptable Approaching Target Not Applicable/ Observable


Candidate fails to exhibit--or exhibits on an inconsistent basis—behaviors associated with a commitment to the profession.

Candidate exhibits most, but not all, behaviors associated with a commitment to the profession.

Candidate consistently and appropriately exhibits behaviors associated with a commitment to the profession.

No opportunity to observe candidate exhibiting behaviors associated with a commitment to the profession.

3. Commitment to Growth: Candidate positively receives, and professionally acts upon, feedback provided by faculty, instructional staff, and/ or supervisors. Candidate remains committed to improving instructional practices and teaching activities. Candidate remains flexible and open to feedback from others.



Candidate fails to exhibit--or exhibits on an inconsistent basis—behaviors associated with a commitment to growth.

Candidate exhibits most, but not all, behaviors associated with a commitment to growth.

Candidate consistently and appropriately exhibits behaviors associated with a commitment to growth.

No opportunity to observe candidate exhibiting behaviors associated with a commitment to growth.

4. Professional Behavior: Candidate maintains a professional appearance and demeanor, in accordance with the standards of the site where he/she is placed. Candidate conveys a positive message about his/her discipline and the teaching profession and is dependable, punctual, and well prepared. Candidate completes assigned tasks, duties and responsibilities on time.



Candidate fails to exhibit--or exhibits on an inconsistent basis— professional behaviors.

Candidate exhibits most, but not all, professional behaviors.

Candidate consistently and appropriately exhibits professional behaviors.

No opportunity to observe candidate exhibiting professional behaviors.

5. Commitment to Student Safety: Candidate acts in a safe and responsible manner, avoiding any action that might put students at physical or emotional risk during all field experiences and related activities (e.g., travel to/from site, service learning experience).

Unacceptable Target Not Applicable/ Observable


Candidate fails to exhibit--or exhibits on an inconsistent basis—behaviors associated with a commitment to student safety.

Candidate consistently and appropriately exhibits behaviors associated with a commitment to student safety.

No opportunity to observe candidate exhibiting behaviors associated with a commitment to student safety.

12

6. Professional Communication: Candidate interacts and communicates in a positive and professional manner with students, peers, school and university personnel, and others. This includes interactions in person and on social media. Candidate communication is free of bias, prejudice, or lack of fairness toward individual student or groups of people.



Candidate fails to exhibit--or exhibits on an inconsistent basis—behaviors associated with professional communication.

Candidate exhibits most, but not all, behaviors associated with professional communication.

Candidate consistently and appropriately exhibits behaviors associated with professional communication.

No opportunity to observe candidate exhibiting behaviors associated with professional communication.

7. Collaboration: Candidate collaborates with others, demonstrates a positive rapport, contributes to group efforts, and shows respect for others and their ideas.



Candidate fails to exhibit--or exhibits on an inconsistent basis—behaviors associated with collaboration.

Candidate exhibits most, but not all, behaviors associated with collaboration.

Candidate consistently and appropriately exhibits behaviors associated with collaboration.

No opportunity to observe candidate exhibiting behaviors associated with collaboration.

8. Commitment to All Learners: Candidate acts and communicates in an effective manner that enhances the educational opportunities for all students and their families. Candidate demonstrates commitment to valuing student diversity in all its forms. Candidate seeks to educate him- or herself and adjust teaching practice when necessary to meet the cultural, academic, social-emotional and other needs of students and their families.



Candidate fails to exhibit--or exhibits on an inconsistent basis—behaviors associated with a commitment to all learners.

Candidate exhibits most, but not all, behaviors associated with a commitment to all learners.

Candidate consistently and appropriately exhibits behaviors associated with a commitment to all learners.

No opportunity to observe candidate exhibiting behaviors associated with a commitment to all learners.

9. Overall Assessment of Candidate Dispositions

Unacceptable Approaching Target Areas for Improvement: Candidate fails to exhibit--or exhibits on an inconsistent basis—a majority of the professional attitudes, values, and beliefs expected as educators interact with students, families, colleagues, and communities.

Candidate exhibits most, but not all, of the professional attitudes, values, and beliefs expected as educators interact with students, families, colleagues, and communities.

Candidate consistently and appropriately exhibits the professional attitudes, values, and beliefs expected as educators interact with students, families, colleagues, and communities.

assessment review panel responses to dispositions rubric 2017 … · teacher education assessment...

Documents