nsta program renewal online workshop. you may have been …nsta reviewer update online training...

1

NSTA Program Renewal online workshop.You may have been a reviewer for awhile, but new things have been happening and changes have been made.http://www.adirondackartistsguild.com

2

NSTA Reviewer Update Online Training

October 12, 2007Erica M. Brownstein

Capital University

NSTA Teacher Accreditation Coordinator

For information contact: [email protected] cell

3

Objectives

• Describe reviewer process

• Explain changes in requirements for NSTA

• Discuss new levels of acceptance for each standard

• Discuss feedback to institutions

• Describe common weaknesses in NSTA program submissions

4

Attachments

• NSTA feedback visual• Sample review• Sample comments to reviewers

Add report form to packet

7

PreviousNSTA / NCATE Alignment

10. Professional Dev.9. Safety & Welfare8. Assessment 7. Science in the Community6. Curriculum5. Skills of Teaching4. Issues in Science3. Inquiry2. Nature of Science

Math in scienceResearch in scienceTechnology in Sc.Unifying ConceptsConcepts & Principles

87654321Standards1.

Sc.

con

tent

If you are familiar with this chart, note the areas that have changed since June of 2007.We have eliminated areas that are common to all programs – to help eliminate redundancy AND let Science Education report focus on items that are unique to science.

8

PresentNSTA / NCATE Alignment

9. Safety & Welfare8. Assessment 7. Science in the Community6. Curriculum4. Issues in Science3. Inquiry2. Nature of Science


87654321Standards1.

Sc.

con

tent

Look at your attachment chart – It should look like this one. This is my B&B when I am reviewing. I keep a printed copy and write on it as I am reviewing.If you are familiar with this chart, note the areas that have changed since June of 2007.We have eliminated areas that are common to all programs – to help eliminate redundancy AND let Science Education report focus on items that are unique to science.

10

Standard 2 using visual

• What is required?– Assessment #3?– Assessment #5?– Assessment #8?

• Does the program address all components?

11

NSTA / NCATE Alignment



87654321Standards1.

Sc.

con

tent

Look at your attachment chart – I look at Standard 2 and realize I will be looking for it in three places – Assessment 3, 5 and 83 is pedagogy5 is student learning8 is contextual content (candidate knowledge and understanding of NOS)

12

Evaluation of each standard

• Met– All areas of standard

have been addressed• Met with conditions

– Some or most areas have been addressed

• Not met– No areas of the

standard have been appropriately addressed

13

Comments for each standard

Remember to:1. have a general

statement summarizing the standard as met, met with conditions or not met

2. include a rationale– each relevant

assessment listed and– feedback for each

assessment

14

Comments for each standard

• Keep in mind – most institutions work

hard to prepare good science teachers.

– Be respectful of their program when making comments.

• As yourself– How would you react

to this comment?

15

Generic SampleStandard XX: Standard

is not met. Rationale• There is no convincing

evidence that assessment XX adequately assesses the important defining elements of the standard. Assessment XX lacks substance and scope that ensures the knowledge needed to (fill in).

16

Comments for reviewsSample comments

– Need to adapt for situation and particular assessment

– Will only cover most common issues

Guidelines for comments– Neutral– Leave out agendas– Only address NSTA 2003

Standards and requirements directly

17

Standard 9Which Assessments?



87654321Standards1.

Sc.

con

tent

Look at safety – it should be in Assessment #4 and #6

18

Standard 9: Assessment 4Safety not specific to science

Standard 9: Standard is met with conditionsRationale• Assessment 4, Student Teaching Observation Form is

required to clearly address all components of Standard 9, 9a, 9b, 9c, 9d. Generic student teaching observation forms are not sufficient to address the specificity needed for safety in the teaching of science.

• Assessment 6 provides convincing evidence that assessment 6 adequately assesses the important defining elements of the standard.

19

Standard 9: Assessment 4

1 2 3 4 5 9b. The candidate practices safe and proper techniques for the preparation, storage, dispensing, supervision, and disposal of all materials used in science instruction.

LikertStandard

1 does not address standard2 addresses but does not meet standard3 meets standard at acceptable level4 above minimum acceptable level5 exceeds standard

Scoring guide needs improvement

20

Standard 9: Standard is met with conditionsRationale• Assessment 4: The assessment addresses the

candidates’ knowledge and understanding of the Standard 9a, 9b, 9c and 9d. However, the criteria do not differentiate different levels of performance in observable ways. It is not clear what actual evidence evaluators will be looking for in order to make judgments about candidate performances for Standards 9a, 9b, 9c, and 9d.

• Assessment 6 provides convincing evidence that the important defining elements of the standard are adequately assessed.

Standard 9: Assessment 4Likert Scale or general categories

21

Standard 9: Assessment 6Restating the standard

Standard: • Know and practice safe and proper techniques for the

preparation, storage, dispensing, supervision, and disposal of all materials used in science instruction.

Criterion: • The candidate knows and practices safe and proper

techniques for the preparation, storage, dispensing, supervision, and disposal of all materials used in science instruction and demonstrates in the classroom.

Scoring guide needs improvement

Focus on measurable (reliable/reliable)

22

Standard 9: Assessment 6Restating the standard

Standard 9 is met with conditions.Rationale• Assessments 4 and 6 should not restate the standard as

the criteria for a given assessment requirement. Standards cannot be used as criteria for judging themselves. Criteria using operational terms are needed.

23

Standard 9: Assessment 6Observable criterion

Standard: • Know and practice safe and proper techniques for the

preparation, storage, dispensing, supervision, and disposal of all materials used in science instruction.

Better Criteria (discrete observable behaviors)In the safety workshop, the candidate

– Labels, stores and dispenses three different chemicals properly

– Uses approved techniques for the handling and preparation of four different chemicals

– Demonstrates three examples of disposing of chemicals in an approved manner

Focus on measurable and observable – can you SEE the criterion?

24

Standard 9: Assessment 6Observable criterion

Standard 9 is met.Rationale• Assessment 4 addresses the candidates’ knowledge and

understanding of the safety for the teaching of science. However, the criteria do not differentiate different levels of performance in observable ways.

• Assessment 6 is aligned with desired outcomes of the standard, has clear performance requirements in place, performance requirements are clearly and adequately described, performance requirements for all dimensions are sufficient to show that candidates are adequately prepared and criteria and minimum performance levels are sufficient.

However, if the program is built on mastery (all candidates must re-do until they master the criteria), then a comment given in #4 is not appropriate. As long as the criteria are observable.

25




87654321Standards1.

Sc.

con

tent

26

Ex: Standard 4 FeedbackNot science specific

Standard 4: Standard is not met. The assessments used to provide evidence for this standard are not science specific therefore they cannot address candidates’ knowledge and ability to understand socially important issues related to science and technology in their field of licensure, as well as processes used to analyze and make decisions on such issues and engage students successfully in the analysis of problems, including considerations of risks, costs, and benefits of alternative solutions; relating these to the knowledge, goals and values of the students.

• Assessment 3 does not address the candidates’ ability to develop lesson plans that address this standard, 4b.

• Assessment 5 does not assess candidates’ ability to engage students successfully in the analysis of problems and relating these to the knowledge, goals, and values of the students (Assessment 5).

• Assessment 8 does not address the candidates’ knowledge and understanding of the issues, Standard 4a. Generally, lesson plans demonstrate pedagogical knowledge but do not demonstrate background knowledge needed to understand socially important issues related to science and technology in their field of licensure, as well as processes used to analyze and make decisions on such issues.

27

Standard 1a: Assessment 2GPA & Content Analysis Forms

• Available on the NCATE website– Upload completed form with the program report

• Align the science content w/ NSTA 2003 Standards

• Must demonstrate a 90% alignment– In each discipline cluster (core, advanced)

• Compare the submitted form with the original on NCATE’s website– # of cells

• Compare the coursework submitted with the required courses– Undergraduate– Post degree– Master’s

28

Assessment #3First question to ask – is it

science specific?– Is the science specificity in

the scoring guide and / or the descriptors?

• If it is not in the scoring guide, then what comment will you use?

• If there are science specific items in the scoring guide,

Second question – Does it have measurable criteria?– See examining

assessments slide

29




87654321Standards1.

Sc.

con

tent

30

Standard 1: Assessment 2Less than 90% alignment w/ Content Analysis Form

Standard 1 is met with conditions.Rationale:• Evidence is provided that candidates understand the most of the

content portion of the standards (Assessments 1 and 2). However, subject matter requirements have not been shown to be 90% aligned with the 2003 NSTA Standards using the Content Analysis Forms.

• Assessment 3 provides evidence that candidates are able to use pedagogy to teach science concepts. Science content requirements for all levels of the program (post degree and master’s) was not included.

• Assessment 5 provides evidence that candidates are able to successfully convey to students science specific concepts.

• Assessment 7 provides convincing evidence of candidates’ ability to do research in science and use mathematics within a science context as presented in the standard (d-e).

• Assessment 8 provides convincing evidence of candidates’ ability to understand 1b and 1c.

31

Sample Assessment #5

• Important assessment• Does it address NSTA Standards 1a, 1c, 2, 3, 4?

– Is the science specificity in the scoring guide and / or the descriptors?

• If it is not in the scoring guide, then what comment will you use?

• If there are science specific items in the scoring guide,

• Second question – Does it have measurable criteria?

32

When examining AssessmentsDoes it meet the 5 things to keep in mind for scoring guides?

1. Do the criteria adequately cover the important defining elements of the desired performance?

2. Do the criteria distinguish the different levels of performance in observable ways?

3. Are the variables in the criteria likely to be interpreted in the same way by different evaluators?

4. Are the criteria just a rehash of the standard or do they define the standard in operational terms?

5. Is there enough substance to the assessment to assure external reviewers that if the criteria are met, the candidate is prepared to teach to the standards?

#1 problem for all SPA’s

33

Common Weaknessesin NSTA program submissions

1. Assessments are not science specific2. Do not include safety in the student teaching observation

form3. Evidence of student learning is 1a only (does not address

1c, 2c, 3b and 4b)4. Content Analysis forms not included, 90% aligned or not all

programs require courses listed5. Assessment 6 lists regular science laboratory classes6. Assessment 7, science research, is a literature review or

just a list of science courses7. Assessment 8 is not contextual content or only meets

some of the required standards.

35

Levels of decisions• Nationally Recognized

– Nationally recognized until next review

– Meets all NSTA standards• Recognized with conditions

– Could be changed in 18 months

• Not recognized– Not science specific– Weak in several areas

• When in doubt – go for “not recognized” and the audit team will re-consider

36

ExampleNational recognition w/ conditions

• Assessment 5 is designed such that candidates provide evidence of k-12 student learning. This would be evidence based, with student scores. The given assessment refers to engaging the students in activities, but there is no attention to the effect of the candidate on student learning. The assessment would need to specifically attend to student learning in the areas of standards 1, 2, 3, 4. The criteria for the scoring guide should be in operational terms.

38

Summary • One thing I’ve learned about reviewing NSTA

programs is . . .• One thing I need to remember is . . .• One thing that surprised me is . . .• I’m going to have to think more about . . .• One way to make this training better fit my

needs is . . .

Please email answers [email protected]

nsta program renewal online workshop. you may have been …nsta reviewer update online training...

Documents