1 3/22/2010 caltpa coordinators’ conference april 11, 2011 9:00 a.m. — 3:00 p.m. california...

13/22/2010

CalTPA Coordinators’ Conference

April 11, 2011

9:00 a.m. — 3:00 p.m.

California Baptist University

23/22/2010

AgendaAgenda9:00 – 9:10 Welcome from CBU and CTC

9:10 – 9:45 New BM and ISC Tasks

9:45 – 10:30

TPA & Accreditation

10:30 – 10:45

Morning Break

10:45 – 11:30

Recalibration

11:30 – 12:30

No - host Lunch in the Cafeteria

12:30 – 1:15

Candidate Feedback

1:15 – 2:00 Remediation

2:00 – 2:15 TPA Data Collection

2:15 – 3:00 Question and Answer Panel

33/22/2010

CalTPA Steering CommitteeCalTPA Steering CommitteeKathy Athey Project IMPACT - SJCOE

Billye Brown Dominican University

Michael Cosenza California Lutheran University

Nedra Crow National University

Ilene Foster Claremont Graduate University

Katie Pedley Educational Testing Service

Mick Verdi CSU San Bernardino

Keith Walters California Baptist University

Sara Cloutier Consultant

Stacy Schmidt CSU Bakersfield

Terry Janicki CTC TPA

Wayne Bacer CTC CalTPA

Mike Taylor CTC Staff

43/22/2010

CalTPA Coordinator Contact Info

• Please check the laptop computer on the sign in table and update your program’s CalTPA Coordinator Contact information

53/22/2010

CalTPA Coordinators’ Conference Feedback Forms

During the conference, use the conference feedback forms to give us constructive feedback on each of the sessions.

Your comments will be instrumental in fine tuning current policy and determining future topics of discussion and direction for the CalTPA

Please hand in at the end of the day

63/22/2010

Since July 2008 we have…

• Built out the SSP task from 5 to 20 different content areas.

• Developed the online CalTPA Coordinators’ resource site that includes assessor training and recalibration resources. (Sparrow)

• Posted a TPA / CalTPA section on the CTC website

• Rewritten the CalTPA Teacher Candidate Handbook

• Developed the CalTPA Implementation Manual

73/22/2010

Since July 2008 Continued…• Initiated the TPA Users Advisory Council and

the CalTPA Steering Committee• Developed the annual TPA data collection

process• Contracted with ETS to provide ongoing

psychometric support• Continued to provide Assessor and Lead

Assessor Training• Held a yearly CalTPA Coordinators’

Conference• Established TPA and CalTPA Listservs• Provided ongoing program support

83/22/2010

CalTPA Online Resources

• CalTPA section of the CTC website– Candidate Handbook– Implementation Manual– Tasks and Rubrics– State Sponsored Assessor Training Calendar – CalPTA Listserv Registration

• CalTPA Coordinators’ Website (Sparrow)– Assessor Training Materials– State Sponsored Recalibration Tasks,

ROEs and Scores

93/22/2010

New CalTPA BM and ISC Tasks

103/22/2010

New CalTPA BM and ISC Tasks Cont...

• New DI and AL Tasks– Validated by the Commission and ETS in

October 2010 – Completed and recently field tested at a

State Sponsored Training

• New SSP and CTE Tasks– Validated by the Commission and ETS in

March 2011 – Are still in the final selection and editing

process

113/22/2010


• DI Score Validation Concerns– He fits the requirement of being an English

learner in a class where there are very few students who do.

– All of the student permission forms along with student and parent names were included in the response.

– Included a class picture.– I like this student personally and I am trying

to use all my resources to help this students succeed. He is smart, but struggles in school due to his language barrier.

123/22/2010


• AL Score Validation Concerns– Student work was not submitted– I selected Mario because he struggles

academically this year. He also seems to have a speech problem.

– The student is a very likeable student with an excellent sense of humor. She has diabetes and misses a great deal of school.

– I selected this student because he has special needs in the classroom. Although he has not yet been tested for any learning disabilities or emotional disorders, he is unable to perform in the classroom without constant adaptations.

133/22/2010


• CTE Score Validation Concerns

– Video was not submitted or would not play

– Student work was not submitted

143/22/2010

New CalTPA BM and ISC Tasks Cont…

• New CTE videos will be posted on Sparrow where they can viewed either online or downloaded as MP4 files

• All new initial calibration case should be posted on Sparrow by July 1st, 2011

153/22/2010

TPA and Accreditation

Terry Janicki, [email protected]

Mike Taylor, Consultant [email protected]

Commission on Teacher Credentialing

163/22/2010

TPA and Accreditation

• Biennial Reports: Assessor data will be submitted

•Program Assessment: Review process for Standards 17-19

•Site Visits: Resources are being developed for use at the site visit

173/22/2010

Biennial Report

Changes to the Biennial Report reports due in Fall 2011 from Past Reports

• The specific information needed is outlined in PSA 10-17 and is related to the assessors used in the implementation of the TPA.

• PSA10-17 http://www.ctc.ca.gov/educator-prep/PS-alerts/2010/PSA-10-17.pdf.

183/22/2010

Changes to Biennial Report

• Number of Assessors • Assessor Initial Training and

Recalibration • Data on Reliability Related to

Double Scoring • Modifications made to assessor

selection, training, recalibration.

193/22/2010

Changes in the Program Assessment Review of Standards 17-19

• A subset of BIR reviewers with particular expertise in the TPA will review the responses to Standards 17-19

• Ensure a fair and rigorous process for the review of Standards 17-19 regardless of TPA model

• Preliminary Findings of Program Assessment reviewers will still be confirmed through interviews and the review of other evidence by BIR members at the site visit

203/22/2010

Guiding Questions in the Program Assessment Review of Standards 17-19

• Questions are not intended to replace the TPA related standards, but rather to guide Program Assessment readers to ask critical, but uniform questions of each program’s response

• Institutions preparing responses may also find these questions helpful as they prepare program assessment documents

213/22/2010

Guiding Questions in the Site Review of Standards 17-19

• In PSA10-17 Appendix A are a listing of

Considerations for Site Visit for Standards 17-19

• Site visit team must interview the individual(s) responsible for the implementation of the TPA. Possible/Likely interview subjects for each component of the standards are indicated in Appendix A

223/22/2010

Short Break

233/22/2010

Calibration & Recalibration

Katie PedleyEducational Testing Service

Ilene Foster Claremont Graduate University

Mick VerdiCSU San Bernardino

243/22/2010

Calibration

• CCSSO & Association of Test Publishers “Operational Best Practices for Statewide Large-Scale Assessment Programs” (2010): – “Processes should be established to

ensure consistent scoring results of open-ended responses….” (14.2)

– “Individual and group rater performance should be measured and analyzed regularly” (14.2.2)

• Calibration is a check that assessors are “on track” prior to scoring

253/22/2010

Why calibrate?

• Established link between calibration and scoring accuracy (Ricker-Pedley, 2010)

• Opportunity to train or refresh on scoring rubric

• A way to help control rater leniency and severity (Congdon & McQueen, 2000; Lumley & McNamara, 1995).

• Creating documentation that assessors met performance standard prior to scoring– In case of legal challenge of scoring

process

263/22/2010

Other Programs

• Praxis, GRE, TOEFL, TOEIC– Initial certification (longer set of cases to

score) – Subsequent calibration prior to EACH

time assessors score—assessors who do not meet a scoring standard are dismissed for the day.

– Significant evidence from these programs that even good, accurate raters can have “off” days.

273/22/2010

Recalibration

GENERAL IDEAS

1. Use Multiple Methods for Recalibration

2. Be Flexible3. Document How and When

Recalibration Occurs4. Check ROEs for details

283/22/2010

Recalibration at CSUSB

• 1. Calibration Every Quarter• Randomly select a percentage of each

assessors and double score• Must have at least 80% agreement and

greater than 50% on point2. Indicator of Validation3. Calibration Meeting4. On line recalibration or retesting5. Retraining and retesting

293/22/2010

Recalibration at CGU

• On-line calibration• Mix previously double scored cases in

the mix of new assessments. Look for consistency of scoring as well as accuracy filling out ROE with specificity.

• The ROE is so important for remediation.

• CGU only assesses a specific Cal TPA once a year (except for resubmissions) and therefore recalibration is needed for everyone at each scoring session.

303/22/2010

Recalibration Table Talk

• Discuss different ways each is recalibrating at their site.

• How are you keeping track of recalibration?

• What type of documentation do you keep?

• What issues or concerns are arising due to recalibration?

313/22/2010

No - host Lunch in the

Cafeteria

323/22/2010

Feedback: Informing Successful TPA Candidates

Michael Cosenza & Nedra Crow

333/22/2010

Purposes

• Provide examples of candidate feedback forms.

• Answer questions.• Discuss participants’ forms.

343/22/2010

CTC & Candidate Feedback

Required: • Overall holistic score.

Suggested: • Rubric that accompanies the score.

Not permitted: • Giving the candidate the ROE.

• Even though the ROE may not be shared with the candidates, additional feedback may be provided.

353/22/2010

Candidate Feedback: Specifics

National:• Provides feedback (passing & non-passing) using:

o TPA’s rubric language.o Specific Task’s sections.

• Feedback transmitted through TaskStream.

Cal Lutheran: • Provides detailed feedback for

remediation but not for those who pass.• Just beginning to provide comments via

a local rubric in TaskStream.

363/22/2010

National University

373/22/2010

California Lutheran University

• Local RubricTPA Task 1 - Subject Specific Pedagogy

Far Below Standard Below Standard Meets Standard Exceeds Standard Score/ Level

Using subject-specific, developmentally appropriate pedagogy.

Inappropriate, irrelevant, inaccurate, or missing.

Minimal, limited, cursory, inconsistent, and/or ambiguous.

Appropriate, relevant, or accurate.

Appropriate, relevant, accurate, and clear or detailed.

Planning for instruction.





Planning for assessment.





Making adaptations.





Overall Comments:____________________________________________________________________________________

383/22/2010

Discussion

• Is your program providing candidate feedback in addition to their score?

• If yes, how does your program provide this feedback?

• Are you using some type of local rubric?

393/22/2010

RemediationKeith Walters

Cal Baptist University

Sara CloutierConsultant

Stacy SchmidtCSU Bakersfield

403/22/2010

Remediation through Independent Study

Candidates register for a 1 unit TPA class for SSP, DI, and AL task

Candidates register for a 2 unit TPA class for the CTE task

Task is due the 7th week of the quarter

Score of 1 or 2 is not passing and requires remediation

413/22/2010


TPA coordinator is available for all candidates

Reviews areas of weaknesses with candidates in a one-on-one setting

Required to revise and resubmit in 10 days

Score of 1 or 2 is not passing

Candidates are not allowed to progress in program

Candidates sent for remediation

423/22/2010


Extended University Course

1-2-3 unit courses

Less expensive

One-on-one remediation

Two more opportunities to pass

433/22/2010

Remediation through Independent StudyCandidates make schedule with TPA

coordinator

10 hours of one-on-one help available per unit

Candidate initiates meetings

443/22/2010

Teach to the test

NOT to the score

453/22/2010

Foundational Abilities – PPT Foundational Abilities – PPT PresentationsPresentations

1. Using Content Specific Pedagogy TPE #1, and #9

2. Learn About Students TPE #3, #6, #7, and #8

3. Creating a Positive Classroom Environment TPE #2, #5, #10, and #11

4. Establishing Learning Goals TPE #1 and #8

5. Plan Instruction and Assessments TPE #3, #4, #6, #8, and #9

6. Make Adaptations TPE #5, #6, #7 and #8

7. Analyzing Student Learning and Lesson Effectiveness TPE #2, #3, and #5

8. Professional Development and Reflection TPE #13

463/22/2010

Student ID#: Multiple Subject Single Subject

Semester Completed: Fall 2009

Check One

Score Level

Level Description

4 The response provides evidence that CLEARLY, CONSISTENTLY, AND CONVINCINGLY demonstrates the

teacher candidate’s ability

The preponderance of evidence provided for each of the domains is APPROPRIATE, RELEVANT, ACCURATE, AND CLEAR OR DETAILED.

Evidence is PURPOSEFULLY CONNECTED AND REINFORCED across the response.

3 The response provides evidence that CLEARLY demonstrates the teacher candidate’s

The preponderance of evidence provided for each of the domains is APPROPRIATE, RELEVANT, OR ACCURATE.

Evidence is CONNECTED across the response.

2 The response provides evidence that PARTIALLY demonstrates the teacher candidate’s ability

The preponderance of evidence provided for each of the domains is MINIMAL, LIMITED, CURSORY, INCONSISTENT, AND / OR AMBIGUOUS.

Evidence is WEAKLY CONNECTED across the response and MAY BE INCONSISTENT.

1 The response provides evidence that DOES LITTLE OR NOTHING demonstrates the teacher candidate’s ability

The preponderance of evidence provided for each of the domains is INAPPROPRIATE, IRRELEVANT, INACCURATE, OR MISSING.

Evidence is UNCONNECTED across the response.

Checked comments indicate focus areas for continued growth:

□ 1. Connecting responses (General)

□ 2. Connecting the video and the written material (General)

□ 3. Coordinating State Academic Content Standards and learning goals (TPE #1)

□ 4. Gathering information on the class and the two focus-students that is not anecdotal (TPE #8)

□ 5. Connecting evidence gathered on students with instructional and adaptation decisions (TPE #9)

□ 6. Managing instructional time for the achievement of instructional goals (TPE #10)

□ 7. Managing routine tasks and lesson transitions for the achievement of instructional goals (TPE #10)

□ 8. Establishing and implementing expectations for academic behavior (TPE #10)

□ 9. Establishing and implementing expectations for social behavior (TPE #11) X 10. Facilitating student participation for achieving the stated instructional goals (TPE #5) X 11. Connecting the lesson plan to the content articulated within the content standards and/or learning goals (TPE #4) X 12. Connecting rationale for instructional strategies to the unique learning needs of the identified class of students (TPE #8) X 13. Connecting rationale for instructional strategies to the identified developmental needs of the students (TPE #6) X 14. Connecting rationale for instructional strategies to the skill level identified within the content standards (TPE #6) X 15. Coordinating rationales with the unique class background and/or prior learning (TPE #8) X 16. Connecting adaptations for the English learning focus student to identified student’s strengths (TPE #7) X 17. Connecting adaptations for the Special Needs focus student to identified student’s strengths (TPE #6)

□ 18. Connecting pedagogical decisions to the unique requirements of the identified content standards (TPE #4)

□ 19. Coordinating assessment plan with the identified developmental level of the State Academic Content Standards (TPE#2)

□ 20. Coordinating assessment plan with the identified developmental level of the students (TPE#2)

□ 21. Analyzing completed assessments in a way that clearly demonstrate student achievement levels (TPE#3)

□ 22. Developing assessment feedback strategies that provide clear information regarding areas where student growth is needed (TPE#3)

□ 23. Connecting reflections back to information provided within the prior steps of the TPA (TPE#13)

• Learn About Learn About Students (#2)Students (#2)

• Making Making Adaptations (#6)Adaptations (#6)

• Planning Planning Instruction (#5)Instruction (#5)

• Content Specific Content Specific Pedagogy (#1)Pedagogy (#1)

• Analyzing Analyzing Student Student Learning (#7)Learning (#7)

AAnalyzingnalyzing

473/22/2010

Break Down

1.1. Connection to Connection to prior prior informationinformation

2.2. Type of Type of questionquestion

3.3. Specific TPE Specific TPE focus – go to focus – go to PPT for helpPPT for help

Positive Positive Environment Environment (#3)(#3)

Analyzing Analyzing Learning (#7)Learning (#7)

Step 2: Learning Environment and Academic Instructional Planning for the Whole Class

A. Information about the Learning Environment

Directions:

Consider what you learned about your students in Step 1. Respond to the

prompts below about the learning environment. For each response include:

a description of what you do to establish and/or maintain an effective environment for learning

a rationale for why the decisions are appropriate for the class.

1. In what ways do you establish and maintain a positive climate for learning?

2. In what ways do you establish and maintain rapport with all students and their families?

3. In what ways do you encourage students to take responsibility for their own learning and to work responsibly with others and independently?

4. In what ways do you establish clear expectations for academic and social behavior and respond to behavior that does not meet those expectations?

.

5. Discuss the classroom routines and procedures that you use, including how you establish and maintain them.

483/22/2010

Confirm

1.1. Address all Address all portions of portions of promptprompt

2.2. Concept Concept embedded embedded in promptin prompt

3.3. Connected Connected clearly and clearly and consistentlconsistently y

Step 2: Learning Environment and Academic Instructional Planning for the Whole Class

A. Information about the Learning Environment

Directions:

Consider what you learned about your students in Step 1. Respond to the prompts below about the learning environment. For each prompt include:

a description of what you do to establish and/or maintain an effective environment for learning, and a rationale for why the decisions are appropriate for the class.

1. In what ways do you establish and maintain a positive climate for learning?

It is extremely important to establish a positive climate within the classroom so that students feel safe and comfortable. If the students don’t feel safe and comfortable then they are not going to be successful in learning and achieving. In this classroom the students are treated fairly and respectfully. This creates a safe learning environment that is based on mutual respect and concern. I relate to the students in a calm and loving manner so they always feel safe and cared for. Students are never put down, insulted, or belittled in my class. They are encouraged to be inquisitive, creative, and involved in their own achievements. This is important because it gives them permission and freedom to actively make choices about their own education. All of these factors affect the classroom environment and shape it to be a positive climate for learning.

2. In what ways do you establish and maintain rapport with all students and their families?

I use many strategies that allow open communication between myself and the students’ parents. The daily homework contracts alert the parents regarding their child’s progress as well as the skills and concepts being taught that week. The contracts have a designated section where parents can write a note voicing any questions or concerns they may have. Because the homework contracts are checked every morning, I am able to respond to any parent notes the following day. This is a highly effective and efficient line of communication. The parents are also encouraged to utilize the school’s email address and phone number to reach me. I also make myself available before and after school for all parents. This is all really important because the parents need to know that they play a huge part in their child’s education and any concerns that they have, need to be addressed quickly. It is extremely important to have parent support so that each child is successful.

493/22/2010

TPA Data Collection

Mike TaylorCTC Consultant

503/22/2010

TPA Data Collection

• 2011 data collection will be for the 09-10 academic year

• Similar process to last year– Excel templates via email

• A few changes– No SSNs– No encrypted email– Templates pre-populated with candidate

names– No candidate scores, only pass/fail

information• Templates distributed at the beginning of May,

due back to CTC end of July

513/22/2010

CalTPAQuestions & Answers

CalTPA Steering Committee

1 3/22/2010 caltpa coordinators’ conference april 11, 2011 9:00 a.m. — 3:00 p.m. california...

Documents