CCLIMATE LIMATE LLITERACY AND ITERACY AND EENERGY NERGY AAWARENESS WARENESS NNETWORK PATHWAYETWORK PATHWAY

SUMMARY OF A RIGOROUS REVIEW PROCESS

CLN Webinar December 7, 2010CLN Webinar December 7, 2010

Outline

CLEAN project and CLEAN collection

Review Process

Results of first review round and gap analysis

Conclusions and Outlook

CLEAN Pathway Project CLEAN Pathway Project GoalsGoals

Stewarding a collection of excellent teaching resources on climate and energy science

Professional and Community Development

Online communities

Facilitate the Climate Literacy Network

CLEAN Pathway: CLEAN Pathway: CollectionCollection

500 excellent digital teaching resources addressing climate science or energy awareness for grades 6-16

Resources scientifically and pedagogically reviewed

Annotations reflect reviewer comments

Resources aligned with Climate Literacy: Essential Principles of Climate Science Energy Awareness Principles National Science Education Standards AAAS Project 2061Benchmarks for Science Literacy NAAEE Excellence in Environmental Edu. Guidelines for

Learning

Framework for collectionFramework for collection

Energy Awareness Principles (newly developed)

A. Earth System Energy B. Human Energy Sources C. Impacts of Energy Use D. Energy Distribution/Costs E. Energy Access and Equity F. Decarbonization Challenges

http://cleanet.org/clean/literacy/energy.html

Every activity included in collection has to address either Climate Literacy: Essential

Principles of Climate Science http://www.climatescience.gov/Library/Literacy/

CLEAN review process

Phase 1 of CLEAN focused on teaching activities

Following phases: broadening of scope to videos, visualizations, lab activities, background materials etc.

Developing Review Criteria

Review criteria specific to teaching activities

Informed by NSDL and SERC guidelines, Merlot criteria, DLESE, Climate Change collection scorecard

Review of e-learning materials requires additional considerations (multi-media elements, navigation etc.)

Climate and energy science evolving science – faster turn-around and need more media to convey

Developing Review Criteria

Initially long list of possible criteria: convergence in phrasing of questions during 7 test review rounds

Test reviews: good agreement for good resources, wide spread in answers for low quality resources

Weekly telecons for resource collectors

What is an excellent What is an excellent activity? activity?

Definition Teaching Activity:Relatively brief set of instructional materials that is presented as a whole, where all the parts and ideas are linked and part of the same activity

Educator should be excited to find this activity when searching for teaching materials.

Avenues into collection

Existing resource pools found through online search

Resources suggested by public through online form

Targeted search after gap analysis

Collaboration with projects that submit resources directlyto CLEAN team (iterations)

Informal vetting

Resource pools assigned to team of 9 collectors

Scanning of resource pools and informal triage

Informal triage: Does this resource meet the CLEAN criteria?

Review questionnaireReview questionnaire

Initial Vetting Review

Scientific accuracy Pedagogic effectiveness Technical quality / Ease of use→ 6-12 questions for each category, overall

rating in rubric format, comment box for annotations

→ Questions help to consider all relevant aspects for each category and lead to overall rating

→ No quantitative, only qualitative recommendation (low – medium – high priority)

Formal vetting step

Basic check for relevance to CLEAN project

Decision: “Will resource likely pass the review?”

Recommendation: Move on to initial review or put in “holding tank”

Review: Scientific accuracy

Considerations for initial science review

Solid, current science

Original data cited and data from a quality source

Attribution

Valid concepts

Supporting references

Science: Use of digitally available scientific data to teach concept

Science: Simplified models of complex Earth system to teach concept

Review: Pedagogic effectiveness Considerations for pedagogic review

Learning objectives

Accommodates diverse learners (learning styles, language, cultural diversity)

Prerequisite skills and understandings

Assessment strategies

Engaging for students in subject and approach

Requires independent/inquiry-based thinking

Pedagogy: Inquiry-based labs or experiments

Pedagogy: Pen-and-paper graphing exercise

Review: Technical quality/ Usability

Considerations for technical/usability review

Ready for use, stands on its own

Clear presentation of content

Software/tools/resources commonly found in classroom

Amount of necessary guidance for students by instructor

Offers comprehensive guide for instructor

Digitally available resource

Ease of use: Printable PDFs with background info and instructions

Ease of use: Clear steps-by-step instruction

Expert science review

External expert with PhD in relevant field reviews scientific quality and accuracy of resource

Activity already passed lower level science review (75 % of resources that passed the CLEAN review were rated scientifically excellent by experts)

Limitation/Challenge: Grade-level appropriate science Difficult to find scientists with enough time

Panel review

Based on NSF-panel review system and AccessData Workshops

Panel provides necessary range of expertise

Teams of 4 educators and scientists review each teaching material based on prior reviews, final decision about inclusion in collection

Comments of all reviewers are compiled into annotation which includes teaching tips

Annotations

All reviewer comments, suggestions and tips are combined in notes to users (annotations)

Annotation draft reviewed during review panels, final clean-up during cataloging process

Annotations add considerable value, insight from scientists or experienced educators

Demo on CLEAN WebsiteDemo on CLEAN Website

http://www.cleanet.org http://www.cleanet.org

Numbers first review cycle

Informal vetting of ~5000 teaching activities

~200 resources passed first review with medium or high priority > forwarded to second review

142 resources passed second review with medium or high rating > forwarded to panel review

94 resources passed panel review (18 passed on to editorial board)

5-7 different people reviewed each resource

Gap Analysis

Holes in collection are apparent – inform targeted search and hopefully future solicitationsTotal # Total #

of of activitiactivitieses

# of # of concepts concepts with no with no matchmatch

# of concepts # of concepts with ≤ 3 with ≤ 3 matchesmatches

Guiding Principle

22 2 5 of 7

EP 1 17 2 3 of 5

EP 2 20 0 3 of 6

EP 3 15 2 3 of 5

EP 4 21 1 2 of 7

EP 5 27 1 3 of 5

EP 6 8 1 5 of 5

EP 7 16 1 4 of 6

Energy Awareness

34 0 1 of 6

Conclusions

Rigorous and transparent review process

Ensures reliable and high-quality resources

Framework of Climate and Energy Literacy Principles allow for gap analysis in collection

Outlook

Targeted search to fill gaps in collection

Broaden scope of collection to other educational resources

Promote the collection and build a community of educators (Professional Development/Discussions)

Refining Energy Literacy framework for collection

Detailed Review Criteria

Details about the CLEAN review process: http://cleanet.org/clean/about/review.html

Link to Initial Vetting Questionnaire:http://cleanet.org/files/clean/about/

clean_vetting_questionnaire.pdf

Link to Review Questionnaire:http://cleanet.org/files/clean/about/

clean_review_questionnaire.pdf

Link to Expert Science Review Questionnaire:

http://cleanet.org/files/clean/about/clean_science_review_questionn.pdf

CLEAN collection: cleanet.org