developing valid and reliable scenario-based assessments - p-12 · pdf file ·...

Developing Valid and ReliableScenario-Based Assessments

New York State Family andConsumer Sciences

1998–2002

Education Development Center, Inc.

55 Chapel Street • Newton, MA 02458-1060 • Phone: 617.969.7100 • www.edc.org

Center forEducation,

Employment andCommunity

Dear Colleagues:

For the last four years, members of our Center for Education, Employment andCommunity team at EDC have collaborated with Family and Consumer Sciences(FACS) teachers and administrators in New York State on the Goals 2000 project tobring scenario-based assessments to the field. What began as an innovative strategyto focus learning on standards, resulted in a state-wide initiative supported andowned by FACS teachers and administrators, and institutionalized within the state’slearning and assessment system.

We have appreciated the opportunity to work with New York State educators on thisimportant and groundbreaking effort. It is a wonderful collaboration that bringsresearch and practice together in ways that make a difference for young people. Wehope that this report will stimulate ideas for everyone who has been involved in theFACS project and others around the country who care deeply about high quality,authentic assessments that tell us how our young people are achieving, and help todesign curricula that matches student needs.

On behalf of the CEEC team working on this project, Clifton Chow, Kerean Grant,Rachel Kimboko, Mya Nelson, we thank the New York State FACS teachers for theopportunity to share this experience and congratulate them on their success.

Yours truly,

Joyce Malyn-SmithSenior Project DirectorCenter for Education, Employment and CommunityEducation Development Center, Inc.

THE STATE EDUCATION DEPARTMENT / THE UNIVERSITY OF THE STATE OF NEW YORK / ALBANY,NY 12234

OFFICE OF WORKFORCE PREPARATION AND CONTINUING EDUCATIONWORKFORCE DEVELOPMENT PROGRAM SUPPORT TEAMCAREER AND TECHNICAL EDUCATION89 WASHINGTON AVE., ROOM 320 EBALBANY, NEW YORK 12234(518) 474-5506; FAX: (518) 474-4494HTTP://WWW.NYSED.GOV/WORKFORCE/

Dear Colleagues:

In 1998, the New York State Education Department (SED) awarded a Statewide Goals 2000

grant to the North Rose Wolcott CSD for the development of intermediate assessments in Home and

Career Skills. These assessments were designed to measure student progress toward achieving the

learning standards in Family and Consumer Sciences and Career Development and Occupational

Studies. During the four years of the grant, scenarios and test questions were written by teachers,

tried out on students, and scored by teachers. This report includes a summary of the activities carried

out under the grant and information on lessons learned from this project. A sample scenario with

questions and scoring guide is also included in this report.

Sincerely,

Marilyn Kucera, NYS FACS Specialist

Mary Oliver, NYS Assessment Advisor

Education Development Center, Inc.Center for Education, Employment, and Community

55 Chapel StreetNewton, MA 02458-1060

617-969-7100 www.edc.org

DEVELOPING VALID AND

RELIABLE SCENARIO-BASED

ASSESSMENTS

New York State Family and Consumer Sciences

1998�2002

Copyright ©2002 by Education Development Center, Inc.

Introduction ..................................................................................................................1

Activities of Years 1–4 ..........................................................................................2

Year 1: Building the Capacity and Validating the Product. ..............................2

Selecting Participants..................................................................................2

Content Validity Matrix..............................................................................2

Developing Capacity ..................................................................................2

Selecting Scenarios to Be Piloted as Assessments..................................3

Establishing Systemic Validity ..................................................................3

Choosing Assessments for the Validation Study ....................................3

Year 2: Developing Reliability. ..............................................................................5

Finalizing Scenarios for PreTesting ..........................................................5

Selecting Participating Sites ......................................................................5

Scoring Process and Analysis....................................................................6

Year 3: Formalizing the Assessment and Item Analysis. ....................................7

Development and Testing of Scenarios and Scoring Guides ................7

Scoring Process and Analysis....................................................................7

Year 4: Institutionalizing the Process and Product. ............................................9

Institutionalizing the Assessment Process and Product

Within the New York state Education Department ................................9

Expanding the Pool of Test Writers ........................................................10

Ensuring Professional Development for Teachers

Through Web-Based Tutorials ................................................................10

Appendix 1 Sample Scenario: “The Jeans That Failed Me” ....................................13

Appendix 2 “The Jeans That Failed Me” Scenario Scoring Guide ........................17

Appendix 3 Excerpts from the Content Topic Matrix for Home and

Career Skills Scenario Development ....................................................................21

Developing Valid and Reliable Scenario-Based Assessments 3

CONTENTS

This report traces the story of a groupof New York State Family andConsumer Sciences (FACS) educatorswho examined scenario-basedassessment that benchmarked theattainment of the New York StateLearning Standards for FACS and theCareer Development and OccupationalStudies (CDOS). It describes theprocess of assessment developmentand provides a detailed glimpse of therigor required to develop valid andreliable tests. It also chronicles theprocess by which a group of class-room teachers became a disciplinedtest development team. This project hasdemonstrated that with appropriatesupport, teachers can develop andimplement strong and reliableassessments that stand up to the rigorrequired by test development experts.

The project’s goals were as follows:

◗ Build the capacity of New York StateFACS teachers to develop and imple-ment valid and reliable scenario basedassessments.

◗ Develop valid and reliable scenario-based assessments.

◗ Evaluate these assessments to deter-mine whether they could be anappropriate assessment vehicle.

Project partners, roles, and contributionswere as follows:

◗ New York State Office of WorkforcePreparation and ContinuingEducation: responsible for overallproject leadership, contributed con-tent expertise, participated in theNational FACS standards initiative,recruited a core group of FACS test

item developers, and developed astate-wide network of FACS teachersand sites

◗ New York State Office of StateAssessment: responsible for allstatewide testing and contributedexpertise and support in developingand implementing statewide assess-ments

◗ North Rose Wolcott School District:responsible for project administrationand budget and contributed projectcoordination and content expertise

◗ Education Development Center, Inc.:responsible for scenario assessmentconceptualization and design andcontributed project planning, initialtraining and assessmentdesign/review, and validity/reliabilityanalyses

◗ New York State FACS teachers: devel-oped scenarios, test questions andscoring guides and helped pretestquestions and score student papers.

The Goals 2000 Statewide Family andConsumer Sciences Project was fundedby the U.S. Office of Education.


INTRODUCTION

LLEESSSSOONN LLEEAARRNNEEDD:: The Family and Consumer Sciences(FACS) community throughout the countryis struggling with how to develop andimplement a type of test that assesses lifeskills in the context of real life.

Selecting Participants In the spring of 1998, the projectcoordinator selected teachers fromfifteen middle schools to participate in

this project. Selection criteria includedthe breadth and depth of Home andCareer Skills program activities,technology readiness, in-service networksupport capability, and administrativecommitment to the initiative. In eachparticipating school, the FACS teachermade a commitment to participate intraining and wrote scenario assessments.These teachers came from schools thatrepresented the geographic, cultural, andsocioeconomic diversity of the state.

Content Validity Matrix

In order to ensure that assessmentproducts reflected both the New YorkState Learning Standards and thecontent of the National FACS Standards,experienced FACS teachers developed amatrix that identified points ofintersection between the two sets ofexpectations.

Developing Capacity

In 1998 the FACS teachers participatedin summer institutes. They becameacquainted with the concept of scenario-based assessment and they learned towrite and edit scenarios. They identifiedcontent topics that reflected both theirinterests and lessons usually taught intheir classrooms. They drafted workplans that identified goals and timelinesfor their assessment writingassignments.

In order to build their capacity as a“learning community,” thegeographically separated team of teacherwriters shared a “virtual workplace”called DocuShare. All participatingeducators and staff were trained atWayne Finger-Lakes BOCES in theDocuShare system (a secure Internet site)and received staff development trainingat Ziff Davis in Rochester, New York.Teachers posted their scenario drafts in aseries of electronic folders on

2 A Case Study: NY FACS 1998–2002

ACTIVITIES OF YEARS 1-4YEAR 1: Building the Capacity andValidating the Product.

LLEESSSSOONN LLEEAARRNNEEDD:: It takes a team of people withdifferent strengths to make anidea reality.

LLEESSSSOONN LLEEAARRNNEEDD:: Strong assessments requireconscious focus on thestandards.

DocuShare. They reviewed scenarios inprogress and provided comments and

suggestions to peers. During the summerand autumn of 1998, the teamdeveloped and posted approximately 25draft scenario assessments.

Selecting Scenarios toBe Piloted asAssessments

The process of identifying whichscenario assessments were to be pilot-tested took place over a four-monthperiod in the fall of 1998. The 25 draftassessments were ranked according tohow well they met the establishedcriteria for validity and the extent towhich they contained technically strongassessment questions and answers. Forall the draft-scenario assessments, theState Education Department worked withexperienced teachers to suggest specificand detailed changes to align theassessments more closely with thecriteria.

Establishing SystemicValidity

Systemic validity provided the means ofevaluating both the process by whichthe New York State FACS scenarios weredeveloped and piloted and the extent towhich their content reflected thestandards as outlined in the standardsmatrix. Systemic validity takes into

account the scope of the scenarios, theirdirectness in measuring the cognitive

skills in the standards and criteriaused in scoring, and the extent towhich the cognitive levelsmeasured in the scenarios reflectthe curriculum and teaching.

Choosing Assessmentsfor the Validation Study

Using the principles and guidelines forevaluating systemic validity, EDC andthe State Education Departmentidentified10 scenarios for inclusion inthe validation study. These 10 scenarioswere developed by teachers who alsoserved as pretesters of scenarios,informing us about issues relating toboth process and content validation.Additional considerations included thefollowing: (1) each scenario representeda distinct FACS topical area; and (2) the10 scenarios were pretested at regionsthroughout the state of New York.

General Validation Strategies EDC performed a gap analysis on eachof the 10 scenarios, evaluating thepresence or absence of the appropriateFACS standard the scenario wasdesigned to measure. Teachers whodeveloped and pretested many of thescenarios included in this reportparticipated in a focus group and aconference call. Their views fed directlyinto the process validation andsupported the content validation. Whereappropriate, EDC analyzed documentsfrom the New York State EducationDepartment to understand and validatethe process by which the scenarios weredeveloped and piloted.


For me, the

biggest

challenges

that first year

were

developing an

understanding

that the whole

scenario

assessment

process has

great value,

and realizing

it is

something

that we can

adopt and

learn to do

well.

LLEESSSSOONN LLEEAARRNNEEDD:: Using a virtual workplacepromotes collaboration but italso entails challenges.

Analyzing Systemic ValidityAnalysis of the systemic validity of theNew York State FACS scenario initiativelooked at content validity and processvalidity. Content area validation refersto a direct evaluation of the presence ofcompetencies and standards from theNew York State FACS and NationalFACS in the scenario instruments.Process area validation refers to a directassessment of the development of thescenarios and how these scenarios werepiloted. An evaluation of 10 scenariosrepresenting 10 distinct FACS topicalareas found that the assessments hadsystemic validity:

◗ A majority (6 of 10) of the scenariosexamined did not contain any gapswith respect to the standards matrix.

◗ A majority of the questions reflectedthe standards they were designed tomeasure. Most of the problems cen-tered around the clarity of the ques-tions.

◗ Evidence indicated that the scenarioswere driven by the FACS curriculum.

Findings about the validity of thescenario development andadministration procedures weresummarized through focus groupmeetings and phone interviews. Theresults were as follows:

◗ All scenario developers followed thestandards matrix as closely as possi-ble, often returning to them to dou-ble-check that the standards werepresent in the scenarios.

◗ No evidence of sample bias wasfound in the gender composition ofstudent test takers. All middle schoolgrade levels were represented.

◗ The developers wanted to make alibrary of exemplars available toteachers. One such exemplary sce-nario was placed on the World WideWeb.

By the end of Year 1, the project hadbuilt a core group of teacher-writerswho had developed a strong sense ofownership of both the scenariodevelopment process and the assessmentproducts. With a reasonable comfortlevel based on the experimental work ofYear 1, and with the continued supportof participating pilot sites and teachers,the project partners embarked on Year 2of the Goals 2000 Statewide FACSProject.


The beauty of

developing

those early

scenarios was

that it forced

us to tie our

curriculum to

the standards,

and set us on

a quest to

create valid,

real-life

situations for

teenagers.

Year 2 focused on reliability testing,improving the scoring process;expanding professional development,and broadening the test sample toinclude more communities across thestate of New York.

Year 2 began with a summer institutewhich provided additional training onwriting scenarios. This instituteexpanded the number of writers anddeepened the knowledge of first-roundwriters. They received in-depthinstruction on methods for developingscenarios, accompanying questions, andscoring guides. A panel of previously

trained peers critiqued the work of newteam members. Increased technicalassistance and in-service training helpedwriters make better use of the virtualworkplace. This additional assistancehelped expedite the work of the writers,editors, and project evaluators.

Finalizing Scenarios forPreTesting

From the end of August to mid Octoberof 1999, writers continued to refine thescenarios based upon input from theeditors, New York State Education

Department personnel, and EDC. TheState Education Department identified12 scenarios as usable for pretesting.

In November, writers continueddeveloping additional scenariosand scoring guides. New YorkState Education Departmentpersonnel shared informationabout the ongoing workthrough in-service trainingsessions and solicited schoolsto serve as pre-test sites.

Selecting ParticipatingSites

In December the AssistantCommissioner, Office of WorkforcePreparation and Continuing Educationsent letters to schools inviting theirparticipation in pretesting of thescenarios. Following New York Stateassessment procedures, pretest bookletsand directions for administering thetests were mailed to 35 schools. Morethan 1,800 students were involved in thepretesting.


YEAR 2: Developing Reliability.

LLEESSSSOONN LLEEAARRNNEEDD:: Scenario assessments helpdeepen teachers’ understandingof what good instruction andassessment are and how theycannot be separated.

LLEESSSSOONN LLEEAARRNNEEDD:: Developing the Scoring Guide isone of the biggest challenges:How to write down in wordsexactly what is wanted as ananswer . . . so other ratersunderstand it.

At first we

thought we

would have to

change the

way we teach

but we quickly

learned that

this is

something we

already do.

Scoring Process andAnalysis

After the pretesting of scenarios, a teamof raters assembled in March to scorethe assessment instruments.Two judges rated each scenariopretest instrument; procedureswere established to ensure thatinter-rater reliability analysiswould not be compromised. Forexample, each rater had noknowledge of the other rater’sscores on the same scenarioinstrument. All scenarios were rated,and 10 were included for analysis.

EDC analyzed the ratings using standardstatistical techniques for reliabilityassessment, with the following results:

◗ Demonstrated inter-rater reliability: 9out of the 10 assessment formsdemonstrated strong inter-rater relia-bility. In most cases, questions ofinter-rater agreement arose fromselected questions in each form.

◗ Demonstrated psychometric validity:Items on the whole measured theknowledge and skills they weredesigned to measure.

Implications for further study includethe following:

◗ Gender differences: Girls consistentlyoutperformed boys in the scenarioassessments; in some cases, this dif-ference was statistically significant atthe 95 percent confidence level.

◗ Additional training: There is evidencethat raters need additional training toimprove reliability in scoring.

By this time, the national FACScommunity had become aware of thework of New York State in developing aformal scenario-based assessment builton the National FACS standards. Interestgrew rapidly. New York State developeda booklet describing the work of theproject and reported its progress duringthe national meeting of the Associationfor Career and Technical Education inDecember 1999. More than 200individuals attended the session.Teachers began to use the New YorkState Education Department web site asa medium to share information.


LLEESSSSOONN LLEEAARRNNEEDD:: With technical assistance andsupport, classroom teachers,drawing upon their richexperience and contentexpertise, can develop valid andreliable scenario-basedassessments.

LLEESSSSOONN LLEEAARRNNEEDD:: Innovators have a responsibilityto share what they have learnedto promote continuousimprovement in their field.

Development andTesting of Scenariosand Scoring Guides

Teachers and writers learned more aboutthe importance of developing clearscoring guidelines. In August andSeptember 2000, the teachers met anddeveloped 20 new scenario assessmentsfor pretesting. From the 20, the StateEducation Department chose 14scenarios forpretesting, andsubsequentlyselected 13 foranalysis.

In December,materials weremailed to 44schools. InJanuary 2001,3,300 studentstook the pretests.

Scoring Process andAnalysis

After the pretesting of scenarios, a teamof teacher-raters assembled in March2001 to score the student work. Theraters were given scoring protocols andguidelines developed by New York Statestaff. Two judges rated each studentbooklet; procedures were established toensure that inter-rater reliabilityanalysis would not be compromised. Asin 2000, raters did not have knowledgeof the other raters’ scores on the samescenario instrument. A total of 13scenarios totaling 2,561 pretest answersheets were scored.

An analysis team at EDC evaluated theprocess by which the New York FACSscenarios were administered to examinewhether there might be procedural biasthat would threaten process validity.They did this by assessing the degree towhich the testing environment wassimilar across sites. The team alsoexamined how closely testinginstructions were followed and whether

there weredisruptions orexcessive noise(environmentalproblems) duringtesting.

There was noevidence that thetestingprocedure, asmeasured by thedifficultiesinstructors and

students had in following testinginstructions, violated the proceduralvalidity of the assessment. The EDCteam concluded that the 2,561 pretestanswer sheets included in this analysismet the most rigorous standards ofprocedural validity.

The EDC team completed an itemanalysis of 13 scenario assessmentsadministered in January 2001 and foundthat all 13 met the most rigorousstandards of psychometric analysis. Theyrecommended keeping 11 forms, withlittle or no revision, as an exemplarybank of scenario assessments. On thewhole, these 13 instruments indicated


The third

year was a

revelation to

me because I

really learned

how to adapt

our scenario

assessments

so they

would be

valid for ALL

students —

the urban,

rural, and

suburban

students who

all have

different

learning

experiences.

YEAR 3: Formalizing the Assessmentand Item Analysis.

that the items function very well indiscriminating the knowledge level ofstudents and in presenting them withsufficiently challenging exercises. These13 forms also demonstrated high levelsof inter-rater reliability.

Significant differences in theperformance of boys and girls werefound. In 2000 this difference wasattributed in some instances to inter-rater disagreement. The analysisteam suggested controlling for inter-rater agreement and furtherexploring this result. In 2002, NewYork State controlled not only forinter-rater agreement, but also fordifferences among schools in rural,urban, and suburban settings; yetthe difference remained. Thereforethe analysis team concluded that thedifferences in the performance of

boys and girls were genuine and not thefunction of random statistical sampling.


Institutionalizing theAssessment Processand Product Within theNew York StateEducation Department

FACS tests were not new to the NewYork State Department Education. Thedepartment had previously producedtests aimed at ensuring minimumproficiency in the FACS skills at thesecondary level. Thus, many of the NewYork State FACS teachers had alreadyparticipated in previous state-ledassessment activities and werecomfortable and familiar with theprocess and products.

In New York as in other states, within astate department of education, testinitiatives move ahead in a variety ofways. Development of a new test orrevision of an old exam is usuallyinitiated by specific fiat of the Board ofRegents and filters down through alllevels of the bureaucracy. In this case,several factors led to theinstitutionalization of the scenario-basedassessments for FACS.

◗ Scenario-based assessment was a newapproach that was of interest to theNew York Office of State Assessment.

◗ The product developed thus far by theGoals 2000 project was credible.

◗ Although development of the scenarioassessment product would be laborintensive, a core of teachers partici-pating in the test developmentprocess was prepared to use this newformat because they felt it was worth-while.

◗ Because the FACS assessment wasdriven by state content standards,moving forward with an official effortwould continue to encourage schoolsto pay more attention to learningstandards.

During Year 4 the Home and CareersSkills scenario assessments were foldedinto the ongoing state test-development

program. The Office of StateAssessment conducted thesample selection; analysis ofdata, and the editing,duplication, and mailing of testmaterials. State procedures andprotocols were followed forassembling pretest and field-

test forms, editing and reviewing items,keyboarding in a standard format,duplicating and shipping the pretest andfield test forms in a secure fashion toparticipating schools, developingauxiliary materials such as answerdocuments and directions foradministering the test, and arranging forthe scoring of nonobjective (subjective)questions such as those accompanyingthe scenario assessment.

Once the assessments were administeredand returned to the New York StateEducation Department Office of StateAssessment, they prepared for the


YEAR 4 Institutionalizing theProcess and Product.

LLEESSSSOONN LLEEAARRNNEEDD:: Without state educationdepartment support, a state leveltest cannot exist.

The teachers

who were

part of this

were excited

and had a

strong sense

of

commitment

and owner-

ship—I think

that’s really

key when

innovating

within a

profession.

scoring sessions. Experienced raters whohad been involved in Goals 2000 fromthe beginning and who had beenthrough the previous two scoringsession were hired as table leaders. FACSteachers were hired as scorers. The tableleaders received two days of trainingusing specific scenarios as prototypesand were assigned to oversee the scoringof three or four scenarios.

At the scoring sessions each table leaderguided six to eight scorers through thescoring process. Table leaders resolvedscoring difficulties, answered questions,and helped resolve differences in theinterpretation of the scoring guide.They randomly “back-scored” tests(scored them a second time) to double-check the results of the scoring at theirtable. They ensured that all papers werescored and recorded according to theprotocols set up by the Office of StateAssessment for scoring subjective(nonobjective) test items in all contentareas. After the scoring session, tableleaders were responsible for revising thescoring guide when they found it to bedeficient or incomplete.

Expanding the Pool ofTest Writers

During Year 4, the project conductedtwo new training sessions, expandingthe pool of item writers by 40. Thesessions focused on prototype scenarios

(“The Jeans that Failed Me,” “The ProjectBox,” “Marco Volunteering”) that hadbeen through field testing analysis,editing and re-editing in Year 2.Training sessions were held inNovember for 20 teachers and inJanuary for another 20.

Teachers were recruited through regionalFACS meetings and through the stateprofessional association. Interestedteachers were invited to the item writertraining sessions. Potential writers camefrom various parts of state. Allparticipants clearly understood that theywere being trained to write testquestions and that they would beassigned items by standard and topic.Each participant made a one yearcommitment to this process. If theirwork showed promise and they wereinterested in continuing as test itemwriters, some of them would be invitedto write more questions, serve onexamination committees, review testpapers, or become raters.

Ensuring ProfessionalDevelopment forTeachers Through Web-Based Tutorials

FACS teachers involved in this projectvoiced concern that in order to befamiliar and comfortable with usingscenarios for both instruction andassessment, teachers would needongoing professional development andsupport. The New York State Office ofWorkforce Preparation and ContinuingEducation produced an online tutorialfor this purpose, through the Goals 2000project in cooperation with the ProjectAccelerate Consortium. The tutorial,“Scenario Assessment for Home andCareer Skills,” is designed to assist


LLEESSSSOONN LLEEAARRNNEEDD:: To maintain a supply of newitems, a continuous effort mustbe made to recruit and train newwriters.

At the end of

the project,

we were

writing high-

level

scenarios and

questions.

We owe this

to the

training we

received and

the

experience of

working

together.

intermediate-level teachers by doing thefollowing:

◗ Reviewing the Intermediate-LevelNew York State Learning Standardsfor Family and Consumer Sciencesand Career Development andOccupational Studies

◗ Reviewing the National Standards forFamily and Consumer SciencesEducation

◗ Showing the content that should beassessed in the mandated Home andCareer Skills program

◗ Providing guidelines for developingscenario assessments

◗ Examining the step-by-step develop-ment of the scenario assessment, “TheJeans that Failed Me.”

FACS educators involved in a projectfocus group reviewed prototype of thetutorial and made revisions. The tutorialwill be reviewed again by the FACSstate in-service teams. Teachers canaccess the tutorial through Accelerate U,a Web-based instructional resourcecenter that offers online professionaldevelopment focusing on New YorkState Learning Standards to educators,students, and the community in NewYork State. More than 19 BOCESs,several city schools, and seven partners,

such as public television station WXXI,participate in Project Accelerate. This“one-stop” comprehensive New YorkState K-12 instructional support Website contains original content, links toWeb resources, and news from educatorsin the field.

The state organization of Family andConsumer Science Educators (FACSE) ispublicizing the tutorial, encouraging itsmembership to view it on the Web site.The free tutorial is available 24 hours aday. FACSE is also using the tutorial totrain teachers on this topic. Educatorscan view the video teaching clips attheir leisure.


We need to

get more

teachers

involved

because this

is such a

great way to

move our

discipline

forward: kids

love these

scenarios and

they really

work as

assessments.LLEESSSSOONN LLEEAARRNNEEDD::

There is a need to inform FACSprofessionals of the importanceof assessing the state learningstandards and to guide them incarrying out effectiveassessments using standards-driven scenarios.

Sample Scenario:“The Jeans That Failed Me”


Appendix 1

1

Michelle is a teenager who feels she is old enough to managethe money she earns from her babysitting jobs. Her parentsare unsure about this.

1. Identify two ways Michelle could prove to her parents that she can effectivelymanage her money. (1 point each for a total of 2 points)

A.

B.

After several months of effectively managing her own money,Michelle decides she wants to buy a pair of jeans. She istrying to decide whether to buy a pair of designer jeans whichare very expensive, or a pair of non-designer jeans which costmuch less.

2. State two reasons why buying a pair of designer jeans might appeal to a teenager. (1 point each for a total of 2 points)

A.

B.

“The Jeans That Failed Me”

PART I

PART II

2

After thinking about which type of jeans to buy, Michelledecided to purchase the designer jeans.

3. List two consumer actions Michelle should take in order to make a wise purchase ofthe designer jeans. (1 point each for a total of 2 points)

A.

B.

Michelle purchased the designer jeans at a local departmentstore. She laundered them according to the care instructionsbut the jeans shrank and no longer fit. Michelle decides toreturn the jeans and goes to the department store where shebought them.

4. List two steps Michelle needs to follow with the sales clerk once she arrives at thestore in order to have a successful return. (1 point each for a total of 2 points)

A.

B.

PART IV

PART III

3

The sales clerk refuses to take the jeans back. Michelle feelsthe clerk is being unfair. Michelle decides to pursue thematter further.

5. State an action Michelle could take and explain how the action could lead to asuccessful return. (2 points)

PART V

“The Jeans That Failed Me”Scenario Scoring Guide


Appendix 2

1

Scoring Guide

1Award one point for each answer that identifies how Michelle could prove to her parents that she caneffectively manage her money.

Examples of one-point answers include but are not limited to:_ develop a money management plan with needs and wants_ prepare a budget and keep track of expenses_ open a savings account and only take out money when she really needs it

Award zero points for each answer that is completely incorrect, irrelevant or incoherent.

[1 POINT EACH FOR A TOTAL OF 2 POINTS]

2Award one point for each answer that states a reason why buying a pair of designer jeans mightappeal to a teenager.

Examples of one-point answers include but are not limited to:_ influence of advertising_ peer pressure_ to help meet ego needs_ designer jeans might be of higher quality_ may have better choice of styles/colors



“The Jeans That Failed Me”

2

Scoring Guide

3Award one point for each answer that lists a consumer action Michelle should take in order to makea wise purchase of the designer jeans.

Examples of one-point answers include but are not limited to:_ compare prices from store to store_ look for sales_ look for quality construction_ compare brands of designer jeans_ try the jeans on in the store to check for proper fit



4Award one point for each answer that lists a step Michelle should follow with the sales clerk.

Examples of one-point answers include but are not limited to:_ approach the clerk in a polite manner_ explain to the clerk the reason for the return_ make sure she brings what she needs (i.e., sales receipt, hang tag)_ inform the clerk what she prefers (to get another pair; refund; credit; or whatever else

store policy allows)



3

Scoring Guide

5 Award two points for an answer that states an action and describes how it could lead to a successful return.

Examples of two-point answers include but are not limited to:_ Explain her situation to the sales manager. He/she has the authority to override the clerk's

refusal and approve the return._ Contact the manufacturer. The manufacturer has an interest in seeing that consumers are

pleased with their purchases.

Award one point for an answer that states an action but does not describe how it could lead to asuccessful return.

Examples of one-point answers include but are not limited to:_ Talk to the sales manager._ Contact the manufacturer.

Award zero points for an answer that is completely incorrect, irrelevant or incoherent.

[TOTAL OF 2 POINTS]

Excerpts from theContent Topic Matrix forHome and Career SkillsScenario Development


Appendix 3

Excerpts from the

Content Topic Matrix forHome and Career SkillsScenario Development


Appendix 3

Home and Career SkillsMod 3—Personal and Family Resource ManagementTopic 1: How Can I Be A Responsible Consumer?

Performance Objective: Identify various influences on student’s individual or family role as a consumer.

◗ Distinguish between the influence of differentindividual and family priorities, needs, wants,values, and lifestyles on consumer decisions

◗ Identify and describe the influence advertisinghas on consumer decisionmaking

◗ Analyze the influence that peers have on con-sumer decisions at different stages of the lifecycle

◗ Determine the influence that availability of theresources of time, effort, money, and skillshave upon consumer decisions

NYS FACS Standards

NYS CDOS Standards

Standard 3—Resource Management

Students will understand and be able to managetheir personal and community resources.

Intermediate Level Family & ConsumerSciences Key Idea 1

Students will understand and be able to managepersonal resources of talent, time, energy, andmoney to make effective decisions in order to bal-ance their obligations to work, family, and self.They will nurture and support positive relationshipsin their homes, workplaces, and communities.They will develop and use their abilities to con-tribute to society through pursuit of a career andcommitment to long-range planning for their per-sonal, professional and academic futures. They willknow and access community resources.

Performance Indicators

◗ Understand how the family can provide for theeconomic, physical, and emotional needs of itsmembers

◗ Understand the resources available, makeinformed decisions about the use of thoseresources and know some ways to expandresources

◗ Understand how working contributes to a qualityliving environment

Standard 3a—Universal Foundation Skills

Students will demonstrate mastery of the founda-tion skills and competencies essential for successin the workplace.

Key Ideas and Intermediate Level PerformanceIndicators

◗ Basic Skills include the abilities to read, write,listen, speak and perform arithmetical andmathematical functions.

◗ Thinking Skills lead to problem solving, experi-menting, and focused observation and allow theapplication of knowledge to new and unfamiliarsituations.

◗ Personal Skills generally include the qualitiesof competence in self-management and abilitiesto plan, organize, and take independent action.

◗ Interpersonal Skills are positive qualities thatlead to teamwork and cooperation in large andsmall groups in family, social, and work situa-tions.

◗ Technology is the process and product ofhuman skill and ingenuity in designing and cre-ating things from available resources to satisfypersonal and societal needs and wants.

continued at right


The chart below is an excerpt from a larger document prepared by the New York State EducationDepartment. The knowledge gained by students using The Jeans That Failed Me scenario is comparedwith three sets of relevant standards: the New York State Family and Consumer Sciences (NYS FACS)Standards, the New York State Career Development and Occupational Studies (NYS CDOS) Standards, andthe FACS National Standards. The complete matrix is available online in PDF format at the website:<http://www.emsc.nysed.gov/workforce/FACSE2/facse.html>.

◗ Consider technology in terms of its relation toa product or service and its impact upon con-sumer decisions

◗ Consider environmental and social issues thatinfringe upon the rights of others in the contextof consumer decisions

NYS CDOS Standards (continued)

FACS NationalStandards

◗ Information management focuses on the abili-ties to access and use information obtainedfrom other people, community resources, andcomputer networks.

◗ Resources management includes the applica-tion of financial and human factors, and the ele-ments of time and materials to successfullycarry out a planned activity.

◗ Systems skills include the understanding of,and ability to work within, natural and construct-ed systems.

Content Standard 2.2

Analyze the relationship of the environment tofamily and consumer resources.

Content Competencies

◗ Examine environmental trends and issuesaffecting families and future generations.

Center for Education,Employment, andCommunity

developing valid and reliable scenario-based assessments - p-12 · pdf file ·...

Documents