measuring progress a consortium workshop on assessing students for technological literacy

Measuring Progress

A Consortium Workshop on Assessing Students for Technological

Literacy

About

Center to Advance the Teaching of Technology & Science

It is the professional development arm of ITEA Initiatives are directed toward:

Development and dissemination of standards-based curriculum materials and supporting resources

Teacher enhancement Research projects Consortium of states

The CATTS Consortium is a cooperative means by which states can efficiently accomplish work that contributes to their respective state plans

Typical Development Process

Identify the course, product, or service Conduct the research Present an outline to the CATTS Consortium Develop the course, product, or service Solicit field reviews Conduct teacher workshops in Consortium states

and/or train trainers at ITEA Conference Use for one year in Consortium states Revise, print, and produce compact disks Disseminate


Standards for Technological Literacy

• Nature of Technology

• Technology and Society

• Design

• Abilities for a Technological World

• The Designed World

A National Model

Elementary• Models for Introducing Technology: A Standards - based Guide

Middle School• Exploring Technology• Invention & Innovation• Technological Systems

High School• Foundations of Technology• Technology Assessment• Issues in Technology• Engineering Design

K

12

Courses Under Development

Engineering Model Course Guide High School

Invention and Innovation Model Course Guide Middle School


Contents

Chapter 1•Learning, Memory, and a Study of TechnologyChapter 2• Designing - From the Beginning• The Natural World• The Human-Made World• Tools & Safety• Engineering Portfolio & Journals• Classroom Store• The Designed WorldChapter 3• Resources


Supported by the National Science Foundation

This Four Project will focus on:• Designing Thematic Units for Grades 5 & 6• Developing Teaching & Learning Resources• Disseminating Workshops & Distance Learning

Design & Innovation Units• Ways Inventions & Innovations are Developed and• How They Affect Us Personally, Socially, & Economically

Technological Systems Units• Ways Systems are Developed, Produced, Controlled, & Assessed• Medical, Agriculture-Biological, Energy-Power, Manufacturing, Communication, Construction

Workshop Objectives

At the end of this workshop, participants will be able to: Effectively assess their students for technological

literacy using standards-based assessment strategies. Apply the results of effective student assessment to

improve teaching and learning and to address issues of accountability.

Teach others about how to effectively assess their students for technological literacy using standards-based assessment strategies.

Professional Development Considerations

Help participants design or select the PDPromote incremental changeProvide for collaborationCreate an atmosphere for open discussionCreate a climate of sustained supportEstablish a rapport were adults have both

permission and expectation to share the responsibility for their learning

NASA Professional Development Benchmark Study (NASA, 2002)

About Adult Learners

Tend to be self-directedHave a rich reservoir of experience that can

serve as a resource for learningHave a life-, task-, or problem-centered

orientation to learning as opposed to a subject matter orientation

Generally motivated to learn due to internal or intrinsic factors


Adult Learners

Used to getting things done on their own

Technologically literateResponsiveFocusedLifelong learnersAmbitiousFearless


Workshop Overview

Introduction to assessing students for technological literacy

What to assessHow to assessHow to gather evidence of learningOrganizing the standards/assessment-

based learning environmentHelping others with the assessment

processAssessing the workshop


Contents

Section One: IntroductionSection Two: Standards, Benchmarks, and Enduring ConceptsSection Three: Assessment, Criteria and Evidence-Gathering ToolsSection Four: Managing the Assessment Process

Notebook Contents

Measuring Progress: Assessing Students for Technological Literacy (Process of Development)

Strategies & Activities, Script AlsoPower Point SlidesStandards for Technological LiteracyState Standards & MaterialsWorksheetsEvaluation

Assessment

Assessment is the systematic, multi-step process of collecting information on student learning, understanding, and capability and using that information to inform instruction and enhance learning.

Assessment

Curriculum assessment and student assessment are different

Curriculum assessment measures the curriculum against frameworks and standards

Student assessment measures student understandings against standards

Why Assess Students for Technological Literacy?

Primarily:To improve students’ learningTo improve teaching effectiveness

And as a result, there are other benefits too!


Advancing Excellence in Technological Literacy:

• Student Assessment• Professional Development• Program Standards

Technology for All Americans Project

Student Assessment STL

Standards-basedClear purposePrinciples of good assessmentWorld beyond school contextsApplication of results

Standards-based means:

Standards for Technological Literacy: Content for the Study of Technology (STL)

There are others, but we’ll discuss those later

Standards-based (STL)

Students will develop an understanding of The Nature of TechnologyStudents will develop an understanding of Technology and SocietyStudents will develop an understanding of

DesignStudents will develop Abilities for a Technological WorldStudents will develop an understanding of The Designed World

Clarity of Purpose

Focus on significant understandings, what is really important to know, be able to do, and truly understand

Establish clearly defined expectations so it will be clear when they are met

Principles of Good Assessment

Assess often Formative Summative

Use a variety of measures Consider learning styles Consider students’ abilities to demonstrate

understandings Match the evidence gathering tool to the

measurement task Use teacher, peer, and self-assessment

Provide timely feedback To students and teachers Not just right or wrong, but how to

improveMaintain security

Establish equity Make expectations clear Treat all students with the same respect Use contexts that have meaning to all

students Be objective; don’t fulfil biased expectations Don’t be predisposed with color, gender,

social class, or the way they talk or look Don’t use assessment as a sorting tool

World Contexts

Couch assessment in everyday situations that are familiar to students(also answers the question of, “yes, but

when am I ever going to use this stuff?”)Be sure that the contexts are

actually familiar to all students

Applications of Results

Communicating achievement and performance to students

Making instructional and program or curriculum decisions

Assigning grades and communicating with parents & guardians

Marketing and promotionCommunicating with other

stakeholders

The Nature of Technology

Students will develop an understanding of The Nature of Technology. This includes acquiring knowledge of:1. The characteristics and scope of

technology.2. The core concepts of technology.3. The relationships among

technologies and the connections between technology and other fields.

Students will develop an understanding of Technology and Society. This includes learning about:4. The cultural, social, economic, and political

effects of technology.5. The effects of technology on the

environment.6. The role of society in the development and

use of technology.7. The influence of technology on history.

Technology and Society

Students will develop an understanding of Design. This includes knowing about: 8. The attributes of design. 9. Engineering design.10. The role of troubleshooting,

research and development, invention and innovation, and experimentation in problem solving.

Design

Students will develop Abilities for a Technological World. This includes becoming able to:11. Apply the design process.12. Use and maintain technological

products and systems.13. Assess the impact of products and

systems.

Abilities for a Technological World

Students will develop an understanding of The Designed World. This includes selecting and using:14. Medical technologies.15. Agricultural and related biotechnologies.16. Energy and power technologies.17. Information and communication

technologies.18. Transportation technologies.19. Manufacturing technologies.20. Construction technologies.

The Designed World

Emotional intelligenceTeam workResponsibilityWriting & mathematics applicationsReadingScience connectionsEconomics, government, globalizationSCANS

Other Important Content, e.g.

SCANS: Skills & Competencies

Competencies Resources Interpersonal Information Systems Technology

Foundation & Personal Qualities Basic Skills Thinking Skills Personal Qualities

A Technological Studies Continuum

Grade Levels

K-2 3-5 6-8 9 10 11 12 13 14 15 16 Graduate|_________________________________________________________________________|

Standards for Technological Literacy (All Levels)

|_________________|Industry Standards (NSSB)

|______________________| Professional Standards (Accreditation)

|________| Professional

Standards for Technological Literacy

•Characteristics & Scope of Technology•Core Concepts of Technology•Relationship of Technology & Other Subjects

•Social, Economic, & Political Effects of Technology•Effects of Technology on the Environment•Role of Society in Technology•Influence of Technology on History

•Attributes of Design•Engineering Design•Trouble Shooting, R&D, Invention & Innovation, Problem Solving

•Apply the Design Process•Use & Maintenance of Technological Systems•Assess Impacts of Technology

•Medical Technologies•Agricultural & Biological Technologies•Energy & Power Technologies•Information & Communication Technologies•Transportation Technologies•Manufacturing Technologies•Construction Technologies

Other Standards

Related to all of Technology•ABET•SCANS•NSSB•Mathematics•Science•English / Language Arts•Social Studies•Etc.

Related to Construction•DPI, DWD, AGC, MCUBC•Milwaukee Area Const. Adv. Comm. •Nat. Center Const. Educ. & Research•Am. Institute of Constructors

Pre - K Life-long Learning

Understanding Defined

Understanding is “the capacity to apply knowledge to new situations in appropriate ways.” (Howard Gardner)

Enduring Concepts: These are: Big (broad with universal

applications) Important (significant) Profound (deep, thoughtful) Lasting (durable through time)Ideas about technology.

So What’s Worthy of Understanding?

“Enduring” understandings (Wiggins & McTighe, 1998):

Represent a big idea having enduring value beyond the classroom.

Reside at the heart of the discipline (involve “doing” the subject).

Require uncoverage (of abstract or often misunderstood ideas).

Offer potential for engaging students.

Enduring Understandings

Understanding

In this book, (Understanding by Design, 1998) we (Wiggins and McTighe) use “understand” to mean that a student has something more than just textbook knowledge and skill – that a student really “gets it.” Understanding involves sophisticated insights and abilities, reflected in varied performances and contexts.

Students need to make conscious sense and apt use of the knowledge they are acquiring as well as the principles underlying it (Wiggins and McTighe (1998).

Wiggins and McTighe (1998) go on to say that understanding involves the abstract and conceptual, not merely the concrete and discrete: concepts, generalizations,

theories, and mental links between facts.

Standards-based & Traditional

Standards-based AssessmentApply Standards for Technological

Literacy Establish expectations/outcomes to

determine what we teachAssess for enduring understandings

that will be lasting through time and apply in different situations

Assess often and provide continuous feedback to students and teachers

Assess students using multiple measures

Assess to improve the learning and teaching process

Assess to improve the Standards for Technological Literacy

Assess to ensure accuracy of content

Traditional AssessmentApply curriculum models that

focus on craft knowledge and skills

Teach our favorite content and then rationalize that we are attaining standards and objectives

Measure for specific content knowledge and craft skills

Assess at the end of a term or when the project is finished

Assess using limited measures such as paper/pencil tests and completed projects

Assess students to determine a grade

Curriculum Development Considerations

Technology Education Delivery

Curriculum (standards-based)

Instruction Assessment

Technology Education Development

Curriculum (standards-based) Assessment Instruction

Measuring Progress: . . .

Standards and Benchmarks - lead to big, important, and

Enduring Concepts - that lead to

Selection of Evidence Gathering Tools

Assessment Criteria - that influence and the design of

Teaching Strategies - that drive the selection of

Learning Activities - that are monitored by

Assessment - that informs

A Standards-based Curriculum Approach with Embedded Assessment

Format

Standard: Engineering DesignBenchmarks: Design Process; Modeling; Testing; Prototyping;

EvaluatingEnduring Concept: Problem-solving/OptimizationSub-concepts: Defining the Challenge Proposing Solutions Testing Possibilities Applying the DesignAssessment Criteria & Levels of Understanding:Learning Activity: Working from a design brief, students will build

a working model of a fish farmEvidence Gathering: Describe, list, sketch, model, brainstorm,

plan, draw,engage experts,etc.

Course or Unit: ________________________________________

Standard(s): __________________________________________

Grade Level: (K-2) (3-5) (6-8) (9-12) ______________________

Enduring Concepts:1. ___________________________ 3. ______________________2. ___________________________ 4. ______________________ An Enduring Concept is a large, important, profound, and lasting idea we want

students to understand. An Understanding is “the capacity to apply knowledge to new situations in

appropriate ways.” (Howard Gardner)

Sub-concepts:1. ___________________________ 3. ______________________2. ___________________________ 4. ______________________

Rubric:Enduring Concept goes here: __________________________

Sub-conceptsLevels

Target3

Draft2

Unacceptable1

Learning Activity with Embedded Assessment (Evidence Gathering):

Activity

Purpose: Introduce STL & State Standards, Objectives, and Materials

Consider: Teachers have students build a computer from component parts, interface the device with input & output devices, and network machines with others

Connect: The learning activity with standards or objectives. Which standards might be addressed?

Activity

Purpose: Introduce Enduring ConceptsConsider: Teachers have students build a computer

from component parts, interface the device with input & output devices, and network machines with others

Connect: Using the benchmarks, connect the learning activity with standards or objectives and enduring concepts. Which standards and enduring concepts might be addressed? What big and important ideas were these teachers addressing?

Purpose: Apply the use of standards, benchmarks, and enduring concepts

Select: A favorite course, identify the standards being addressed, and using the benchmarks, identify the enduring concepts.

Select: One enduring concept to be assessedWorksheet: Blue worksheet in notebook

Activity (Part 2)

Problem Solving/Optimization

Element

Levels

Defining theChallenge

Proposing Solutions Testing Possibilities Applying the Design

3The challenge is veryclear, specific, andfocused

Sketches, drawings, anddescriptions suggest awide range ofpossibilities

Effective drawings,descriptions, andprototypes of severalpossibilities

An effective drawing,model, portfolio, andpresentation of acreative and workablesolution

2The challenge issomewhat specific andfocused

Sketches, drawings, anddescriptions show some“out-of-the-box”possibilities

Somewhat effectivedrawings and prototypesof a few possibilities

A reasonably effectivedrawing, model,portfolio, andpresentation of a some-what creative solution

1The challenge needs tobe more clearly statedand focused

Proposals suggest alimited number ofpossibilities

Minimally effectivedrawing and one prototypeof a possible solution

A drawing, model,portfolio, andpresentation of asolution

Activity

Purpose: Write assessment criteriaUsing: The standard and selected

enduring conceptReduce: The enduring concept into 3 -

6 component parts (sub-concepts)Enter: Those sub-concepts across the

top of the rubric

Enduring Concept

Sub-

LevelSub-concept Sub-concept Sub-concept Sub-concept

3

2

1

Enduring Concept

Sub-concepts

Levels

Sub-concept Sub-concept Sub-concept Sub-concept

3

2

1

Assessment Criteria and Evidence Gathering

Assessment criteria help to determine if a student understands

Evidence is how a student demonstrates understanding, e.g., performs on a test

In all cases, make it clear to the student what is expected and measure accordingly

Embed the assessment into learning activities

Problem Solving/Optimization

Element

Levels

Defining theChallenge

Proposing Solutions Testing Possibilities Applying the Design

3The challenge is veryclear, specific, andfocused

Sketches, drawings, anddescriptions suggest awide range ofpossibilities

Effective drawings,descriptions, andprototypes of severalpossibilities

An effective drawing,model, portfolio, andpresentation of acreative and workablesolution

2The challenge issomewhat specific andfocused

Sketches, drawings, anddescriptions show some“out-of-the-box”possibilities

Somewhat effectivedrawings and prototypesof a few possibilities

A reasonably effectivedrawing, model,portfolio, andpresentation of a some-what creative solution

1The challenge needs tobe more clearly statedand focused

Proposals suggest alimited number ofpossibilities

Minimally effectivedrawing and one prototypeof a possible solution

A drawing, model,portfolio, andpresentation of asolution

Grades 9-1220 Standards and Corresponding Benchmarks

24 (estimated) Enduring ConceptsSorted into Compatible Groups then Labeled as Courses

Group of 6Enduring Concepts




x six x sixx six x six

OR

Engineering Course

List the Standards being AddressedSix (estimated) Enduring Concepts18-24 Sub-concepts

Technology Program

Course Examples

Middle School Model Courses Exploring Technology Invention & Innovation Technology Systems

High School Model Courses Foundations of Technology Technology Assessment Issues in Technology Engineering

Teaching Strategy

Problem solvingModeling and prototypingRealizations/products/projectsCooperative learningModular teachingOut-of-school learning

Measuring Progress

A Consortium Workshop on Assessing Students for Technological

Literacy

Bismarck, North DakotaMarch 28 & 29, 2003

Course or Unit: ________________________________________

Standard(s): __________________________________________

Grade Level: (K-2) (3-5) (6-8) (9-12) ______________________




Sub-concepts:1. ___________________________ 3. ______________________2. ___________________________ 4. ______________________


Sub-conceptsLevels

Target3

Draft2

Unacceptable1


Learning Activities

Have meaning and purposeAre selected or developed after assessment

criteria have been writtenEnable students to experience content that

will lead to understanding important concepts that are driven by standards

Enable students to “do the subject,” “do technology”

Have assessment embedded whenever possible


Digital Libraryfor Technological

Literacy

It’s more than just another Web-site!

National Digital Library for Technological Literacy

Innovation Curriculum Online Resources

International Technology Education Association

Eisenhower National Clearinghouse

Innovation Curriculum Online Resources

NSF digital library collection Manages technological literacy resourcesInformed by Standards for Technological

LiteracyAvailable Winter 2002

Technology: A Definition

“Know-how that extends human capability”

A common misconception is that technology is only computers, networking, and the like

Standards for Technological Literacy:Content for the Study of Technology

Categories:Nature of TechnologyTechnology and SocietyDesignAbilities for a Technological WorldThe Designed World

Published by ITEASupported by NSF & NASAApproved by NRCEndorsed by NAE

NDLTL Project Goals

Create an electronic system to connect users with technological literacy resources.

Monitor and maintain the digital library collection.

Build and sustain the digital library.Promote digital library tools and

services to relevant professional communities.

Standards-basedMetadata

Grade LevelsK-2, 3-5, 6-8, 9-12

Nature of Technology

Technology & Society

Design

Technological Systems

Resources Essential toTechnology

Resources CurriculumCoursesActivitiesUnits

MediaTextsInternet linksGraphicsSoftware

Collection Matrix

Status of the Project

Currently loading the system using selection criteria

Pilot within the next month followed by field testing

Available January 2003Operational Fall 2003

Evidence Gathering Tools

Portfolios and JournalsScenariosOpen-ended QuestioningModels and PrototypesRealizations/Products/ProjectsObservationsDiscussions/InterviewsConcept MappingPresentations and DebatesPaper/pencil Tests

Portfolios & Journals

Similar in that they are collections of information (evidence), usually at the end of a major activity or unit

Different in that: Portfolios are collects of physical

objects, but do include writings Journals are written logs of daily or

weekly progress on a significant project

Scenarios

Stories about possibilitiesBased on data, trends, interactions,

and futures

Open-ended Questioning

Formats (Freedman, 1994) Analysis (examine variables of a situation) Comparison (examine similarities and

differences) Description (explain or report) Evaluation (collect & analyze data and make

judgments) Fiction (stories) Problem solving (define, experiment, conclude)

Models and Prototypes

Models tend to be: Idea generators Examples of possibilities Communication tools

Prototypes tend to be: Examples of a final solution

Realizations/Products/Projects

Why not just call it what it is, a project

Turning ideas into realizations Concept drawing or description Prototypes System or process to do something

Embedding assessment

Observation

Observation is seeing and moreWorks well in a laboratory setting

where process is importantUse a form to record observations

Large and small group student discussions

Teachers ask questions of students as they work

Students interview resource persons

Discussion/Interview

Concept Map

1 2b

a A B

C

Issue

Concept Mapping Example

Issue: Air pollution in an urban area as a result of heavy automotive traffic. With this being the case, then what?

Fuel efficient vehicles

Hybrid Vehicles

Mass transit

Smaller, lighter vehicles

Fuel efficient engines

Safety is compromised

More commuters choose to drive

If this, then what?

Universal Systems Model

Resources

Input Process Output

Feedback

Systems Example

Input: Need for electrical energyResource Input: An appropriate fuel, e.g.,

fossil fuel (coal)Process: Use coal to heat water to produce

steam to drive a turbine to turn a generator to produce electricity

Output: Electricity, new problems & challenges, potentially new knowledge

Feedback: Considering the outputs, what adjustments need to be made?

Tests

Paper/pencil tests still have a placeThey usually include multiple choice

and true-falseAlternative assessment usually means

alternative to paper/pencil testingAuthentic usually mean that the test

is presented in a context that exists outside of school

Course or Unit: ________________________________________

Standard(s): __________________________________________

Grade Level: (K-2) (3-5) (6-8) (9-12) ______________________




Sub-concepts:1. ___________________________ 3. ______________________2. ___________________________ 4. ______________________


Sub-conceptsLevels

Target3

Draft2

Unacceptable1


Creating an Equitable Environment

Assess against criteria, use a variety of tools, do it often, and provide feedback

Make expectations clear to studentsTreat students objectively, regardless of color,

gender, social class, or the way they talk or look

Treat all students with equal seriousnessProvide accommodations as necessaryUse out-of-school contexts that are familiar to

students

Applying Assessment Results

Improving learning and teachingReporting student achievementEvaluating program effectivenessCommunicating with stakeholdersInfluencing state and national

assessments

Assessing the Assessment Process

Using Student Assessment Standards for Technological Literacy

Student assessment is consistent with STLThe assessment purpose is clearly definedThe assessment process applies accepted

assessment principlesAssessment is authentic and contextualAssessment results are applied

measuring progress a consortium workshop on assessing students for technological literacy

Documents

technological world

consortium workshop

consortium statesrevise

dissemination of standards

based curriculum materials

natural world

thematic units

designed worldchapter