American Diploma Project

Research Progress Report for

Kentucky P-16 Council MeetingDecember 2, 2002

ADP States

• Indiana• Kentucky• Massachusetts• Nevada• Texas

ADP Partner Organizations• Achieve, Inc.

www.achieve.org

• Education Trustwww.edtrust.org

• Thomas B. Fordham Foundationwww.edexcellence.net

• National Alliance of Businesswww.nab.com

ADP GoalMake the high school diploma

meaningful• Secure demand for standards-based assessment data from postsecondary institutions and employers

• Help states ensure that they have set the bar for high school graduation in reading, writing and mathematics at the right level.

• Establish new benchmarks in reading, writing and mathematics that all states can use

ADP Partner State Commitments

• Partner state leaders* have agreed to rally behind current standards-based systems and to participate in a review of their current standards and assessments

• Business and postsecondary leaders in partner states have agreed to consider standards-based assessment data in admissions, placement and hiring decisions

*Governors, chiefs, postsecondary executives, business leaders

ADP Project Commitments

• Conduct Research

• Provide Technical Assistance

• Develop New Benchmarks

Research

– Legal Study: assisting partner states with new frontiers; legal implications of a K-16 system (Nixon-Peabody)

– Defining Workplace Expectations: Defining the reading, writing and math skills needed for success in new economy occupations (ETS and NAB)

– Defining Postsecondary Expectations: Defining the reading, writing and math skills needed for success in postsecondary institutions (Ed Trust and Achieve)

Legal StudyStandards Reform, Federal Law

and the American Diploma Project: A Framework for Making legally Sound

DecisionsNixon-Peabody LLP

• Prospective Legal Analysis of:– Relevant federal laws– Ways that ADP implicates federal law

• Provides:– Framework for states for analyzing

ADP efforts– Recommendations for minimizing risk

Relevant Federal Laws

• Due Process• Discrimination

– As applied to Education– As applied to Employment Practices

• Tests serve different purposes in different contexts– Looking back (K-12)– Looking forward (admissions & placement)– Implications for validity and opportunity to

learn

• Deference often given to education judgments

Re: GI Forum v. Texas Education Agency

• Despite sobering differences in pass rates, the standards-based high school exit exam – provided an objective assessment of mastery of a

discrete set of skills and knowledge linked to the state’s educational standards,

– served to motivate students and – provided support for systemic accountability.

• TAAS met currently accepted standards for:– curricular validity– opportunity to learn the matter covered by the test– remediation efforts– multiple opportunities to pass

“ Central to the court’s reasoning on the due process and discrimination claims was its belief that the TAAS was ultimately educationally beneficial to all students, and that the state was using assessment data to address rather than exacerbate, disparities in educational opportunity and achievement.”

Recommendations

• Identify key employment and postsecondary expectations

• Conduct gap analysis and align standards for high school exit, college and employment

• Ensure that curriculum, instruction and interventions align to standards

• Assess student knowledge of standards and ensure predictive validity of assessments

• Phase in appropriate use of high school assessment data in admissions, placement and hiring decisions

Defining Workplace ExpectationsETS, NAB & Project Staff

• Defined “good jobs”– Higher wages (top two tiers of

pyramid, 25- 40+K/year)– Insurance and retirement benefits– Opportunities for additional education

or training

• Examined labor market projections• Tracked course-taking patterns• Developed preliminary benchmarks

Distribution of Education & Jobs: 1973 v. 2000

• Percentage of employed who were high school dropouts/had not attended college

1973: 72% 2000: 41% • Percentage of employed who had

some college or a B.A.

1973: 21% 2000: 48%

Employment Pyramid 2000-20102000-2010Projected Growth Rate: 20%Net New Jobs: 7.5 millionTotal Job Openings: 14.5 million (51% from job creation)

2000-2010Projected Growth Rate: 15%Net New Jobs: 8.1 millionTotal Job Openings: 25.2 million (32% from job creation)

2000-2010Projected Growth Rate: 12% (White Collar: 15%, Blue Collar: 10%)

Net New Jobs: 6.6 million (White Collar: 4.2m, Blue Collar: 2.4m)

Total Job Openings: 17.9 million (37% from job creation)(White Collar: 10m, 42% from new jobs, Blue Collar: 7.9m, 18% fromnew jobs)

Jobs Include: Managers; Engineers; Legal Professionals;Licensed Medical Professionals, Teachers, Financial,Insurance, and Real Estate Professionals, TechnicalKnowledge Workers

•Blue-Collar Jobs Include: Protective Services; CraftsWorkers; Mechanics, Repairers, and Service Technicians

Jobs Include: Clerical, Cashiers and Retail SalesWorkers; Personal Services; Food Services; ChildCare Services; Health and Recreation Services;Laborers; Transportation Operatives; Farming,Forestry, and Fishing

Source: Author’s analysis of Bureau of Labor Statistics Employment Projections, 2000-2010; Current Population Survey, March 2001.

Highly Paid

Professional Jobs

Well-paid

Skilled Jobs

Earnings: $25,000 - $40,000

Share of Jobs: 37%

Low-paid or

Low-skilled

Jobs

Earnings: Less than $25,000

Share of Jobs: 38%

Earnings:

$40,000+

Share of Jobs:25%

Figure 2: Employment Pyramid

15-25% of LP/LS Jobs are Transitional(Share of All Jobs: 6-11%)

•White-Collar Jobs Include: Financial Services Support,Administrative Support, Health Technicians; Human Services;Sales Managers;

Workplace Study Highlights

• Algebra II is the benchmark course for students aspiring to highly paid professional jobs or well-paid, white-collar jobs.

• Geometry is the benchmark course for students intending to work in well-paid, blue-collar jobs and low-paid/low-skilled jobs.

• Four years of English that is at least at grade level is a benchmark requirement in the vast majority of jobs.

Preliminary Workplace Benchmarks

• Based on course-taking patterns identified in study

• Two panels of curricular experts refined and revised benchmarks

• Benchmarks circulated by NAB among employers in occupations identified by ETS

• Consistency with postsecondary benchmarks

• Flatter (leaner) management structures– are less dependent on manual labor;– require more skilled workers who can identify and

solve problems, offer innovative ideas for improving productivity;

– demand that workers can articulate clearly, think creatively, absorb data, read and write effectively;

– expect that workers can draw on a stronger foundation of skills

– prompt workers to cope with constant change

• Computers render jobs obsolete; raise stakes for workers who use them

Employers supported preliminary benchmarks

• Suggested students apply knowledge and skills in real-world situations w/o becoming “narrowly vocational”

• Encouraged interdisciplinary learning• Reiterated importance of strong reading &

writing skills for success in teams • Strongly encouraged better teacher

training• Endorsed benchmarks (rigorous curricula)

that will keep more students in the pipeline for postsecondary degrees and promotability

Employers’ suggested areas of emphasis: ELA

• Analytical thinking (v. “interpretive” thinking): develop, analyze, present a logical argument, in writing and orally; summarize & synthesize information; extract essential information; evaluate the validity of information

• Use correct English grammar and sentence structure

• Speak persuasively under time constraints• Translate technical information to a non-

technical audience• Plan and complete extended research

projects (research as part of problem-solving)

Employers’ suggested areas of emphasis: math

• Budgeting, accounting• Probability and statistics• Ability to express real-world problems in

mathematical terms• Ability to understand and apply

mathematics in new and unfamiliar contexts

• Metric system• Formulating arguments (e.g., proofs)• Data interpretation

Defining Postsecondary Expectations

Education Trust, Achieve, Project Staff• Test Content Analysis (January ‘02)• Examination of state standards• Examination of alignment between

standards & assessments• Cross-content area faculty defined

reading, writing and math competencies necessary for postsecondary success

• Cross-content area faculty discussed: “To what extent do current state standards & assessments reflect postsecondary expectations?”

Cross-State Postsecondary Expectations:

• Consistency across systems within states, especially in reading & writing– Some consistency across content

areas

• Consistency across states

• Consistency with workplace expectations

Team findings (Reading):

• Postsecondary expectations are not fully addressed in the Kentucky Core Content

• Standards should include critical thinking skills specifically w/regard to reading

• Standards should emphasize analytic analyses (v. personal reflections)

• Quality and complexity of reading should be delineated

• Some standards too vague to understand actual student expectations

Team findings (Writing):

• Postsecondary expectations are not fully addressed in the Kentucky Core Content

• Clarify jargon• Specify thesis and idea development• Require analytic writing, development

of arguments, compare and contrast essays

• Writing standards should be genre-specific

Team findings (Math):

• Math core content contains most of postsecondary expectations for non-math-dependent fields, but not for math-dependent

• Standards should include dimensional analysis, bases of number systems, developing proofs, translating between metric and other measurement systems, understanding formulas

• Organization (concepts v. skills v. relationships) is confusing

• Standards should clarify calculator policy

Team findings: Reading Assessment

• Greater emphasis needed on higher order comprehension skills and persuasive text analysis

• Content centrality: 80% alignment, but 20% could not be determined because standards are too broadly-worded

• Performance centrality: only 60 % alignment

• Level of cognitive demand too low (86% of items at levels 1 and 2.

Team findings: Writing Assessment

• Holistic scoring guide useful; would serve as strong foundation for genre-specific scoring guides

• Samples of proficient responses less than proficient for postsecondary admissions or placement

• Move on-demand writing to fall of 12th grade

• Correct grammar & usage, analytic and persuasive writing should be emphasized

Team findings: Math Assessment

• Greater emphasis needed on algebraic ideas, concepts, skills and relationships (currently only three of 19 algebra objectives are assessed); also more on geometry

• Reduce percentage of assessment dedicated to probability and statistics

• Content centrality: only 43.3% alignment• Performance Centrality: only 46.7% alignment• Most important standards are left unassessed • Level of cognitive demand too low (90% at

levels 1 and 2

ADP State Policy Panels

• Review Research Findings

• Consider policy options for meeting ADP goals– Use of standards-based assessment data in

high school graduation, college admissions/placement and hiring decisions

• Develop plan for meeting ADP goals and closing gaps between current standards/assessments and postsecondary and employer expectations over time

Kentucky Policy Issues

• Current high school assessments do not reflect most important content in the standards (as prioritized by postsecondary faculty and employers)

• Some important content is missing from standards and assessments

• Assessments do not generate individual scores or reports by domain & item-type

• Timing of assessments makes data difficult to use

Short-term steps for revising reading & writing standards &

assessments • Raise cognitive demand of items in bank• Revise writing anchors to reflect

postsecondary expectations for placement • Revise rubric to be genre-specific• Add analytical writing to writing portfolio• Clarify calculator and formula sheet policies• Revisit mathematics scoring guide• Consider use of KEMPT to supplement KCCT• Pilot alternative admissions, placement and

hiring policies

Long-term considerations• Matrix testing v. individual student (NCLB

requirements)• Streamline and revisit content of POS,

expectations and core content• Clarify quality and complexity of reading in

standards• Revisit timing of tests; consider adding end-

of-course assessments• Design predictive validity studies• Revise admissions, placement and hiring

policies• Report scores on transcripts• Default curriculum and interventions (OTL)

ADP Phase II: New Benchmarks

• Work with partner state policy panels to implement ADP goals

• Synthesize workplace and postsecondary expectations (to create draft benchmarks)

• Circulate draft benchmarks• Supervise content area expert panels• Establish National Advisory Panel• Conduct campaign encouraging use of

benchmarks by all states