Using International Assessment data to Evaluate Educational Systems The PISA 2009 Results-Implications for Trinidad and Tobago

Using International Assessment data to Evaluate Educational SystemsThe PISA 2009 Results-Implications for Trinidad and TobagoJEROME DE LISLESCHOOL OF EDUCATION, UWI, ST AUGUSTINEHARRILAL SEECHARANMANAGER OF INTERNATIONAL ASSESSMENTS & ACTING DIRECTOR OF DERE, TRINIDAD AND TOBAGO MINISTRY OF EDUCATIONScheduleIntroduction to International Assessments and Evaluating Education Systems -Jerome De Lisle (15 minutes)Questions & Comments (5 minutes)Presentation-2009 PISA Results - Harrilal Seecharan (40 Minutes)Questions & Comments (15 minutes)Using the data-A Research Agenda for the SOE (15 minutes)Questions & Comments (10 minutes)What I would Like to know and do-Audience Activity/Discussion (20 minutes)Trinidad and Tobago & International AssessmentsTrinidad and Tobago is the only country in the Caribbean to participate in international assessments.Part of the Vision 2020 strategy-it allows judgment on comparative equity and performance of the education system.Allows benchmarking against other Latin American economies and also that of the WestPIRLS 2006

5

6Performance Labels T & T National Tests(2005)International Benchmarks in Reading (PIRLS 2006)Level 4 (Exceed Standards)Students performing at this level consistently demonstrate ability to proofread for errors and misspelled words. They demonstrate mastery of knowledge and skills needed to infer meanings from context and analysis of word parts. They consistently provide evidence of literal, inferential, and interpretative understanding of texts. They demonstrate mastery of parts of speech and punctuation and spelling in different contextsAdvancedStudents performing this level respond fully to the PIRLS 2006 assessment. They could make interpretations of figurative language and demonstrate that they understand the function of organizational features. They can integrate information across the texts, and provide full text-based support. They comprehend, interpret, and integrate details across.HighStudents reaching this level are competent readers. They could retrieve significant details embedded across the text and provides text-based support for inferences. They could use organizational features to navigate through the informational texts, and make inferences and connections. Students recognize main ideas, some textual features and elements, and are beginning to integrate ideas and information across texts.Level 3 (Meets Standards)Students usually demonstrate the ability to proofread errors and mis-spelled words. They demonstrate mastery of most knowledge and skills needed to infer word meanings from context and analysis of word parts. Their responses usually provide evidence of inferential and interpretative understanding of texts and indicate the ability to follow instructions and to apply appropriate comprehension strategies. IntermediateStudents demonstrate some reading proficiency, especially with the stories. They are able to understand the plots at a literal level, and also to make some inferences and connections across the texts. In the informational texts, they are able to use text organizers (headings, illustrations, etc) to find information beyond the initial parts of the texts, and to provide two pieces of information in answering a question.Level 2Students occasionally demonstrate the ability to proofread for errors and correct mis-spelled words. They demonstrate master of some skills needed to infer word meanings from context and analyze words parts. Their responses occasionally provide evidence of literal, inferential, and interpretative understanding of texts. Challenging questions are answered directly from texts. The student can sometimes locate, select and apply information and use punctuation and capitalization correctly.LowStudents display basic reading skills. They are able to recognize, locate and reproduce explicitly stated details from the informational texts, particularly if the details were close to the beginning of the text. They also demonstrate success with some items requiring straightforward inferences.

8

9Benchmarking Trinidad & Tobago in PISANation ClassificationNationsDifferentiation ScoreInternational AssessmentTarget NationTrinidad & TobagoNAPIRLS 2006PISA 2009High Performing, Non-DifferentiatedCanada, BCNASweden5USANALow Performing, Non-DifferentiatedPoland4Iceland8Norway1High Performing, DifferentiatedGermany19England/UK21Low Performing, DifferentiatedBelgium (French)23Slovak Republic20Oil Based EconomiesQatarNAIranNAHigh Performing AsianSingaporeNAHong KongNALow Performing, AsianIndonesiaNALatin American EconomiesArgentinaNABrazilNAChileNAMexicoNAPanamaNAPeruNAUruguayNA10Benchmarking-Comparing Systems and Outcomes

1112PISA 2009

15

ReadingMathematicsScienceGender GapGender GapGender Gap16Benchmarking Trinidad & Tobago in PISANation ClassificationNationsDifferentiation ScoreInternational AssessmentTarget NationTrinidad & TobagoNAPIRLS 2006PISA 2009High Performing, Non-DifferentiatedCanada, BCNASweden5USANALow Performing, Non-DifferentiatedPoland4Iceland8Norway1High Performing, DifferentiatedGermany19England/UK21Low Performing, DifferentiatedBelgium (French)23Slovak Republic20Oil Based EconomiesQatarNAIranNAHigh Performing AsianSingaporeNAHong KongNALow Performing, AsianIndonesiaNALatin American EconomiesArgentinaNABrazilNAChileNAMexicoNAPanamaNAPeruNAUruguayNA17

2021Last scheduled assessment is PIRLS 20112012May 14164th-- Reading Development Group meeting to conduct scale anchoring of achievement data (Sweden)June 24298th --National Research Coordinators meeting to review draft International Reporttext, graphics, and tables September--PIRLS 2011 Encyclopedia published and posted on web December 11--TIMSS & PIRLS International Study Center/IEA conduct international press conference to release International Report 2013February 7--TIMSS & PIRLS International Study Center distributes final PIRLS2011 International Database and User Guide to countries February 1015--9th National Research Coordinator meeting to conduct training in use of PIRLS 2011 International Database

Trinidads involvement in PIRLS 2011PIRLS 2011 is the third cycle of IEAs Progress in International Reading Literacy Study(PIRLS). Building on the highly successful implementation of its predecessors, PIRLS 2001 and PIRLS2006, PIRLS2011 collects data to provide information on trends in reading literacy achievement of fourth-grade students, while providing baseline data for new countries. Combining newly developed reading assessment passages and questions for 2011 with a selection of secure assessment passages and questions from 2001 and 2006, PIRLS 2011 offers a state-of-the-art assessment of reading comprehension that allows measurement of change since 2001, and includes a full complement of questionnaires to investigate the experiences young children have at home and school in learning to read. AcronymsTIMSS-Trends in International Mathematics and SciencePIRLS-Progress in International ReadingPISA-Programme for International Student AssessmentLLECE Laboratorio Latin americano de Evalucaion de la Calidad de la Educacion MLA Monitoring Learning Achievement PASEC Programme of Educational Systems Analysis SACMEQ Southern African Consortium for Monitoring Educational Quality

Evaluating Educational SystemsNational Evaluation Systems are designed to measure and judge the effectiveness of educational systems, including targeted reforms.Include the use of data from national assessments, regional national assessments (MLA, LLECE, PASEQ, & SAM, and international assessments (PISA, PIRLS, TIMMS).The nature of the assessmentsUsually include both measures of achievement and survey instruments designed to measure factors associated with learning.Tremendous growth in national assessments associated with progress towards the millennium goals and increasing participation in international assessments. Limitations and Challenges of international assessmentsEvaluating quality across countries and regions complex for international agencies and costly for small countries like Trinidad and Tobago.Much progress is due to advances in the technology of testing and sampling.What is to be measured?Measures of Achievement often relate to key measureable skills related to human capital development in industrialized countries.Reading, Mathematics, and Science are critical but there are also international surveys of ICT.What is really measured?Whether skills of achievement based on curricula the target of measurement are quality of the education system and the degree of equity.Equity is measured by (1) the size of an inequality on a basic skill or (2) the relationship between an extraneous variable and that skillIndices of MeasurementBoth norm referenced data (mean scale scores etc) and stdanrds referenced data (Achievement levels based on specified benchmarks) are provided.Standard referenced data is critical for evaluators helping to answer the question, how good is good enough.Insights offered by both types of data may differ greatly.

Comparing International AssessmentsDiffer in intent and information provided. Several countries like QATAR do all three assessments. PISA is unusually strong in analytical work and policy development.

USING THE DATAComparisons & BenchmarkingCan we compare education systems on quality and equity?Are there unique relationship between system individual and school characteristics and quality and equity?How do systems compare with others since human capital development is a central tenet of modern development theory.How OECD countries use dataBenchmarking and comparison studies are commonUsed to discover relationship between structure (differentiation) and equityUsed to study specific populations such as students below the benchmarkUsed to validate national assessment resultsUsed to judge impact of reforms and validity of claims-high standards?International Assessments, Research, Media, and Education PolicyInternational assessments can have a strong impact on the research field, media and national policy. Two examples are the impact of PISA in Germany and the impact of PIRLS 2006 on education policy in South Africa.Naumann, J. (2005). TIMSS, PISA, PIRLS and low educational achievement in world society. Prospects, XXXV(2), 229248.

Understanding PISAThe students tested by PISA are aged between 15 years and 3 months and 16 years and 2 months at the beginning of the assessment period. The school year pupils are in is not taken into consideration. Only students at school are tested. In PISA 2006 , however, several countries also used a grade-based sample of students. This made it possible also to study how age and school year interact.

Understanding PISAEach student takes a two-hour handwritten test. Part of the test is multiple-choice and part involves fuller answers. In total there are six and a half hours of assessment material, but each student is not tested on all the parts. Participating students also answer a questionnaire on their background including learning habits, motivation and family. School directors also fill in a questionnaire describing school demographics, funding etc.

A Brief HistoryDeveloped from 1997, the first PISA assessment was carried out in 2000. The tests are taken every three years. The process of seeing through a single PISA cycle, start-to-finish, takes over 4 years. Every period of assessment specializes in one particular subject, but also tests the other main areas studied. The subject specialization is rotated through each PISA cycle. In 2000, 265 000 students from 32 countries took part in PISA; 28 of them were OECD member countries. In 2002 the same tests were taken by 11 more "partner" countries (i.e. non-OECD members). The main focus of the 2000 tests was reading literacyA Brief HistoryOver 275 000 students took part in PISA 2003, which was conducted in 41 countries, including all 30 OECD countries. The focus was mathematics literacy, testing real-life situations in which mathematics is useful. Problem solving was also tested for the first time.In 2006, 57 countries participated, and the main focus of PISA 2006 was science literacy.

As in PIRLS, definitions are criticalIs the definition in the measurement instruments similar to that enacted in the curricula

Using the dataA Research Agenda for the SOEJerome De LisleUsing Secondary dataThe international assessment databases provide high quality data sets that can be used to understand the nature of educational achievement and school performance.This is secondary data, information which are collected by others in another context and used by an investigator for his own purpose.

Thesis Work Using Secondary DataSeveral high quality thesis have been generated using secondary data-The focus has been on an issue and the focus is on comparisons. Here is a selection:Nanoyama, Y. A cross-national, multilevel study of family background and school resource effects on student achievement. Unpublished Ph.D. Thesis, New York: Columbia University.Park, H. (2005). Cross-national variation in the effects of family background and schools on student achievement: The role of institutional and policy contexts. Unpublished Ph.D. Thesis. Madison: University of Wisconsin: Madison.Powell, T. A. (2007). Families, schools, and national contexts: The effects of institutions and inequality on educational achievement across industrialized nations. Unpublished Ph.D. Thesis. Durham: Duke University.Yang, Y. (2003). Measuring socioeconomic status and its effects at individual and collective levels: A cross-country comparison. Gothenburg: Acta Universitatis Gothonburgenis, Gothenburg Studies in Educational Science 193.

CapacityThe data is readily available, but requires two critical capacities-Collaborative networks and statistical skills.Researchers usually work together on large scale data projects sharing different insights. It is not easy to manage large data sets. Local researchers can team up with international groups. Skills in linear and logistic regression, multilevel and structural equation modeling are most useful in mining the data. Opportunities for Qualitative ResearchersLarge scale data usually highlights specific issues that must be addressed by more detailed case studies.Qualitative researchers can work with quantitative researchers and use mixed method sampling designs to select schools to study.Developing Our Interest in EquityNATIONAL ASSESSMENTSINTERNATIONAL ASSESSMENTSNational Assessment Data first made the situation apparent in 2005. Validated by PIRLS 2006 and PISA 2009STUDIES ON EQUITYQUAL Studies on Disadvantaged Groups (Mechanisms)QUAN Studies using secondary dataQUAN Studies using primary data (Mechanisms)What I Would Like to Know and Do-> Audience Activity/ Discussion (20 minutes)InstructionsWorking in your group (READING, MATHEMATICS, SCIENCE) decide how the group might proceed developing capacity in the use of PISA data. Provide a timeline for the selected tasks.PossibilitiesGather Reading MaterialMine PISA/ PIRLS DatabasesDevelop skills in MM & SEWork with International ColleaguesStudy Issues QualitativelyFoster conversations with media/publicDevelop Evidence-based PolicyAdvocate for the continued use of international assessmentsVideo Analysis of High Performing SystemsOntario CanadaKey Personnel-Ben Levin (Former Deputy Minister of Education)Mary Jean Gallagher

Comments?