optimizing assessment for all - brookings institution...optimizing assessment for all may 2020...

OPTIMIZING ASSESSMENT FOR ALLMAY 2020

OPTIMIZING ASSESSMENT FOR ALLClassroom-based assessments of 21st century skills in theDemocratic Republic of Congo, The Gambia, and Zambia

Authors Helyn Kim was a fellow in the Global Economy and Development Program at theBrookings Institution Esther Care is a senior fellow in the Global Economy and Development Program at theBrookings Institution

Optimizing Assessment for All (OAA) is a project of the Brookings Institution. The aim ofOAA is to support countries to improve the assessment, teaching, and learning of 21stcentury skills through increasing assessment literacy among regional and nationaleducation stakeholders, focusing on the constructive use of assessment in education,and developing new methods for assessing 21st century skills.

Acknowledgements The authors and the OAA Africa National Teams appreciate the support of UNESCO’sTeaching and Learning Educators’ Network for Transformation (TALENT) and ourthanks are extended in particular to Davide Ruscelli. The authors gratefullyacknowledge the Democratic Republic of Congo National Team; The Gambia NationalTeam; and the Zambia National Team for their substantial contributions to theirrespective country sections of the report. Additionally, the authors thank Joesal Jan A.Marabe of the Assessment Curriculum and Technology Research Centre, University ofthe Philippines Diliman, for technical contributions, and Aynur Gul Sahin for hercreative and editorial support. The Brookings Institution is a nonprofit organization devoted to independent researchand policy solutions. Its mission is to conduct high-quality, independent research and,based on that research, to provide innovative, practical recommendations forpolicymakers and the public. The conclusions and recommendations of any Brookingspublication are solely those of its author(s), and do not reflect the views of theInstitution, its management, or its other scholars. Brookings gratefully acknowledgesthe support provided by Porticus. Brookings recognizes that the value it provides is in its commitment to quality,independence, and impact. Activities supported by its donors reflect this commitment. Cover photo credit: St. Peter's Lower Basic School, Banjul, The Gambia

OPTIMIZING ASSESSMENT FOR ALL

PAGE 2

TABLE OF CONTENTS


PAGE 3

EXECUTIVE STATEMENTINTRODUCTIONTHE OAA APPROACH FOR AFRICATHE THREE FOCUS COUNTRIES

Democratic Republic of CongoThe GambiaZambia

OAA PROCESS FOR TASK DEVELOPMENT WORKSHOP 1UNDERSTANDING THE NATURE OF SKILLS

The Gambia: Approach to understanding skillsTHE TARGET SKILLS: PROBLEM SOLVING AND COLLABORATION

The Gambia: Understanding problem solving and collaborationRE-VISIONING EXISTING ITEMS TO TARGET 21CSThink Aloud Sessions

The Gambia: Think aloudGENERATING NEW 21CS ITEMS BASED ON TEMPLATES WORKSHOP 2TASK AND ITEM DEVELOPMENT APPROACHES

Paneling of tasks: Nationals and collaborative WORKSHOP 3PILOTING OF TASKS

DRC: Pilot of TasksKEY POINTS WORKSHOP 4WHAT DO THE PILOT DATA TELL US?THE PILOT DETAILDESCRIPTION OF THE TASKSANALYSISTASK REVIEW CONCLUSIONREFERENCESAPPENDIX A: NATIONAL TEAMSAPPENDIX B: THINK ALOUD GUIDELINES

"Think Aloud" Record FormAPPENDIX C: TEMPLATES AND EXAMPLES

04050507 09 10 12 1315 17 18 19 22 2323242427 28293031 34

EXECUTIVE STATEMENTThe Optimizing Assessment for All(OAA) project at Brookings exploresapproaches to educational assessment,specifically through developingassessments of 21st century skills(21CS). Twenty-first century skills are now firmlyensconced as new learning goals ineducation systems worldwide, but theirimplementation in teaching andassessment practices is lagging behind. We have taken decades to understandhow to teach mainstream educationsubjects like mathematics, history,science, and language. But with thesenew learning goals, which prioritize howto go about getting answers, rather thanjust providing a correct response, we arefacing new challenges in bothassessment and teaching. If we canidentify useful approaches toassessment of 21CS in the classroom,then both the assessment toolsthemselves as well as how studentsengage with them can provide insightsfor teaching the skills. The overall goal of the OAA project is tostrengthen systems’ capacity to integrate21CS into their teaching and learningusing assessment as the entry point tochanging education practices inalignment with changing learning goals.Specifically, OAA is designed to shiftmind and practices around the use ofassessment; shift perceptions on howassessment relates to the broadereducation structure—that assessment isnot something separate but a critical partof the learning and teaching process;and develop new methods for assessing21CS in the classroom.

OAA, in collaboration with participatingcountries from Asia and Africa, hashelped identify 21CS valued by thesecountries, hypothesized what theseskills might look like in classroomassessment tasks, and developed thesetasks with teachers to ensure that theyare usable and valuable in theclassrooms. Notably, OAA has workedwith established approaches toassessment with which teachers arefamiliar, and adjusted them to reflectnew learning goals. Of course, the workgoes far beyond assessment toimplications for how we think abouteducation and what we value in theclassroom. What we value are thethinking and social processes thatindividuals use to explore andunderstand their environment. More comprehensive information aboutthe complete OAA approach can befound in the “Optimizing Assessment forAll: Framework” report, while in thisreport we focus on the collaborativeactivities undertaken in Africa by theDemocratic Republic of Congo, TheGambia, and Zambia to create 21CSassessment tasks. The mechanics of theactivities are described in detail in orderto illustrate the methods used in theproject and by the countries. Foradditional examples and guides for taskcreation, see our forthcoming fourthreport, and for discussion about scalingand implementing the OAA approach,see the forthcoming fifth report in theseries.


PAGE 4

INTRODUCTIONOne of the main goals of OAA was forparticipating countries to developclassroom assessment tasks that canmeasure 21CS. The project adopted acollaborative approach to developcapacity in assessment design. Theproject was structured so that nationalteams had the opportunity to developassessment tasks together at theregional level, as well as individually atthe national level. The objective was toensure that the national teams wereconfident in the usability of thedeveloped tasks and in their ability tocontinue to develop tasks for theirparticular conditions, needs, andcurriculum. The development processwas undertaken through a series ofworkshops, usually convened in one ofthe participating countries so that thenational teams had the opportunity tounderstand the conditions under whicheach was working. Between workshops,in-country development work took place,both within the national teams, and withteachers from participating schools ineach country. The collaborativeapproach and task developmentprocesses are described in this report—covering the workshops and in-countryactivities—and culminating in a pilot ofthe assessments across the countries. As education systems increasinglyemphasize the need for students toapply their learning, focus on 21CS hasintensified. With relatively littleassessment of 21CS in education, OAAhas taken the stance that assessment inthe classroom will provide the supportneeded for teaching in the classroom.Development of classroom assessmentswill provide grounded examples of whatstudent learning practices indemonstration of 21CS look like, thusinforming the development of tools foruse at regional and national levels.

Similar to the work with three countriesin Asia, the Brookings Institution workedintensively with three countries in Africa:two Anglophone countries (The Gambiaand Zambia) and one Francophonecountry (Democratic Republic of Congo[DRC]). The activities were undertakenover a 15-month period through the OAAinitiative to improve the assessment,teaching, and learning of 21CS, withsupport from the Teaching and LearningEducators’ Network for Transformation(TALENT) at UNESCO Dakar. Althoughthe overall objectives of the OAAinitiative in Africa are identical to thoseof Asia, a slightly different approach wastaken. The decision to modify theapproach was both strategic andpractical. The Africa approach drew onlessons learned both from the Asiaexperience (since OAA began in Africaabout six months after Asia), as well asfrom a “mini-study” involving nineAfrican countries that identified currentnational- and classroom-based tests ortest items that would capture 21CS.

THE OAA APPROACHFOR AFRICA


PAGE 5

Lead National Team members from The Gambia, DRC, andSenegal with OAA Brookings scholars convening in Dakar,Senegal to discuss the project

As countries increasingly adopt learninggoals that reflect more holisticperspectives on education, they engagein curriculum review and consideration ofwhat other aspects of education deliveryneed modification. The nature oflearning goals has implications forselecting different forms of assessment,in that some learning goals, such asmemorization of facts, for example, caneasily be captured through constrainedforms of assessment (e.g., multiplechoice, short answer questions,or “fill inthe gaps” responses). Other learninggoals associated with knowledge orskills that are not easily demonstratedthrough the written word might morecoherently be captured through lessconstrained assessment types. Theseinclude tasks that require multiple steps,which might be taken in differentsequences or that might require workingwith different media. In OAA, the focus ison how we might capture studentproficiencies in 21CS in the classroom.Of course, to ensure alignmentthroughout the system, the forms ofclassroom assessment also need to beconsistent with large-scale assessment.The latter typically relies on constrainedforms of assessment since these arerelatively easy to standardize andefficient to administer. OAA is concernedwith optimizing the links between theefficiencies of large-scale assessmentforms and the potential richness ofclassroom-based assessment. The ideaof “vertical coherence” (Herman, 2010),where assessment at all levels of theeducation system is aligned withlearning goals, underpins the OAAmodel. The approach for Africa: 1) capitalizedon the outcomes of the OAA Africa andAsia mini-studies, specifically thepotential for adapting some existingtools to assess 21CS; 2) incorporatedlearnings from the OAA three-countrywork in Asia; and 3) ensured that theactivities and outputs in the two regionswere complementary rather thanreplicated.

This approach enabled the explorationof multiple approaches to designing,developing, and using classroom-basedassessments of 21CS. The National Technical Teams in OAAAsia were most strongly influenced byassessment personnel, while the teamsfor Africa included stronger teacher andpedagogical expertise representation.Members of each of the Africa teamsrepresented assessment, curriculum,and pedagogical expertise, and allteams included a minimum of oneteacher. Therefore, the key differencesbetween OAA Africa and OAA Asiaapproaches are: greater teacherinvolvement at the team level in Africa(as distinct from the teacher involvementwithin participating schools in bothregions); and tool adaptation in Africarather than development from ground upas in Asia. The suite of tools anddevelopment of an assessment guide(described in the forthcoming fourthreport in the series) from the two regionsare complementary. Common to bothregions are general principles andprocesses of item and test development,scoring, and targeting of both constructsand student abilities.


PAGE 6

OAA-participating countries

THE THREE FOCUSCOUNTRIESThe DRC, The Gambia, and Zambiaworked collaboratively to design,develop, and use 21CS assessments.These three countries were identifiedduring the Africa mini-study based ontheir commitment to integrating 21CSinto their education systems. Forinformation regarding the NationalTeams, see Appendix A.

DEMOCRATIC REPUBLIC OFCONGO (DRC)In its most recent country-wide reformagenda, the DRC identified education asone of the most important drivers inaddressing the country’s resourcemanagement gap. The government'svision for the education sector is "theconstruction of an inclusive and qualityeducation system that contributeseffectively to national development, thepromotion of peace and activedemocratic citizenship" by equippingCongolese students with 21CS,such as creativity, critical thinking,problem solving, and the ability to takeinitiative.

OAA Africa National Technical Teams from the Democratic Republic of Congo, The Gambia and Zambia at the first workshop in Banjul,The Gambia.


PAGE 7

Participating countries explored as acollaborative group what was alreadyfamiliar in order to proceed to the nextsteps in learning. Practically, thismeant that some existing assessmenttools collected through the mini-studywere adapted and extended totarget 21CS, as well as developingnew tools.It takes a range of assessmenttechniques to measure 21CS:Available assessment examples fromthe mini-study included short answeritems, essays, and set tasks. Usingthese as starting points, templateswere developed, which providedframeworks for developing new itemsand scoring approaches for 21CS.It is collaborative but reflects eachcountry's unique curriculum: Thenational teams worked together onskill descriptions, assessment itemtypes, and item review but unlike inAsia, the teams were not required todevelop items that target identicalcurricular topics across countries.

In general, the Africa approach ischaracterized by the following:

The education system aims at theharmonious formation of the Congoleseman, responsible citizen, useful tohimself and to the society able topromote the development of the countryand the national culture. DRC is focusedon training productive, creative, cultured,conscientious, free and responsiblecitizens, open to social, cultural andaesthetic, spiritual and republican values(Education Charter, CNS, 1992).

THE GAMBIA

Non-discriminatory and all-inclusiveprovision of education, in particularwith respect to gender equity andtargeting of the poor and thedisadvantaged groups;Respect for the rights of the individual,cultural diversity, indigenouslanguages, and knowledge;Promotion of ethical norms and valuesand a culture of peace; andDevelopment of science andtechnology competencies for thedesired quantum leap.

The Gambia is committed to developingits human resource base, with prioritygiven to free basic education for allthrough “accessible, equitable andinclusive quality education forsustainable development.” As highlightedin Education Policy 2016-2030, theguiding principles of the education sectorare:

The education sector aims to ensure thatteaching and learning focus ondeveloping the physical and mental skillswhich will contribute to nation building—economically, socially, and culturally—and develop creativity and the analyticalmind. In keeping with the country’scommitment to the SustainableDevelopment Goals, the education sectoris dedicated to promoting life skillseducation to help learners acquire notonly knowledge and skills but alsoadaptive and positive behaviors in achanging social and economicenvironment.

ZAMBIA

Self-motivated, life-long learners; Confident and productive individuals ;andHolistic, independent learners with thevalues, skills, and knowledge toenable success in school and in life.

Strive for personal excellence; Build positive relationships withothers; Become good citizens; and Celebrate their faith and respect thediversity of beliefs of others.

The goal of the education system inZambia is to “nurture the holisticdevelopment of all individuals and topromote the social and economic welfareof society.” Zambia’s vision focuses onproviding “quality and relevant lifelongeducation and skills training for all,” thatis accessible, inclusive, and relevant toindividual, national, and global valuesystem s. Zambia is committed toproviding an education that will meet theneeds of Zambia and its people. Theaims of the Zambian curriculum are totrain:

Learners acquire a set of values, which encourage them to:

In addition, the Education CurriculumFramework 2013 identifies keycompetencies for learners at each levelof school that go beyond literacy andnumeracy skills to include critical,analytic, strategic, and creative thinking;problem solving; self management;relationship skills; civic competence;participation; and teamwork.


PAGE 8

The OAA Africa approach values theprocesses and learning that took placeas team members engaged in the seriesof task development activities, ratherthan the set of tools developed duringthat process. The latter are relevant onlyinsofar as they are a testament to thecapacity of the National Teams and theirunderstanding of test developmentaligned with local use. Over the courseof 15 months, the countries engaged infour workshops that built upon eachother, as well as in-country activitiesbetween workshops, which weredesigned to apply the concepts learnedin the workshops to schools andclassrooms, and with teachers andstudents. For example, National Teammembers engaged with schools andpolicymakers to generate buy-in andbuild practitioner understanding around21CS. They visited schools tounderstand the classroom contexts. Theyalso conducted teacher training sessionsto improve teachers’ understanding of21CS; increase their ability to identify theskills demonstrated by students in theclassroom; develop new 21CS items thatcapture the skills that they can use intheir classrooms; and troubleshootscoring issues associated with theassessment of student behaviors. The purpose of OAA is therefore notabout generating a tool that can beused widely but rather to provide aprototype or a model approach thatcountries can use to integrate 21CSinto their teaching and learning. Figure1 shows the workshop series andregional convenings (in large bluecircles), as well as the in-countryactivities.

Figure 1. Series of activities for theOAA task development process

1

2

3

4


PAGE 9

OAA PROCESS FOR TASK DEVELOPMENT

UNDERSTANDING THE NATURE OF SKILLS1

Understand the nature of skills and theimplications for assessmentdevelopment; Select, define, and deconstruct thetarget skills;Re-vision existing assessment itemsto target 21CS;Conduct “think aloud” sessions toverify the skills and their componentsbeing prompted by the items;Generate new items that target skillsat different levels of difficulty; Panel items; Pilot items in schools;Analyze student responses; andReview assessment tasks.

The steps of the process through theworkshops and in-country schedulesinclude:


PAGE 10

Enthusiasm from Gambian school children

Education systems around the worldacknowledge the importance of 21CS,such as problem solving, collaboration,and critical thinking. However, if the goalis to teach and learn these skills, merelyidentifying which skills are consideredimportant or even defining the skills isnot enough. More in-depthunderstanding of the nature of the skillsis necessary (Care, Kim, Vista, &Anderson, 2018). The defining characteristics of a 21CSadopted in OAA is that an individual orgroup of individuals can bring thatcompetency to bear in and across newsituations. Skills are different fromknowledge in that, although knowledgecan be acquired, that in and of itself isnot sufficient to put that knowledge intopractice. Skills enable one to applyknowledge to different situations andtransfer what has been learned in onecontext to another.

This means that actual skills recognitionis important. Traditional methods ofinformation dissemination are notenough to facilitate the application andtransfer of skills to new or differentsituations. Authentic learning tasks (i.e.,tasks that are similar to the ones thatstudents will face in the real world) canprovide opportunities to apply the skillsin different ways.

School walls in The Gambia not limiting horizons

The Gambia: Approach tounderstanding skills For the Ministry of Basic andSecondary Education (MoBSE) in theGambia, providing quality educationfor all is a core mandate. Therefore,promoting the use of 21CS inclassrooms to enhance effectiveteaching, promote independentlearning, and reshape the existingassessment system is a major goal.Although 21CS are highly valued, acloser examination of the curriculummaterials revealed that 21CS,specifically problem solving andcollaboration, “accidentally”appeared on only a few occasions. Inother words, there was no deliberateattempt to integrate the skills withinteaching and learning.Understanding the nature of skillswas one of the most criticalcomponents of this OAA projectbecause it is the foundation for notonly teaching and assessing 21CSbut also integrating it into theeducation system. The first in-country activity that the NationalTeam members conducted was aday-long training session withrepresentatives of four pilot schoolsto discuss the OAA project, exposethem to 21CS, and discuss how theskills can be used to enhanceeffective teaching and learning in theclassroom and beyond. The schoolswere Mansa Colley Bojang LowerBasic School (LBS), St. Peter’s LBS,Abuko LBS, and St. Mary’s LBS.Three teachers from each of the fourpilot schools (in total, three femaleand nine male) attended.


PAGE 11

Inadequate or poor planning forcollaborative activities that canhinder the progress of a lesson;Lack of space that can make itdifficult to carry out activities; Lack of time for certain activitiesto be completed in a class periodthat is usually 30-35 minutes; andLack of availability of materialsrequired for some activities atschools.

The activity was designed to raiseteacher awareness as well asdevelop understanding about the useof the 21CS, and consider how tostimulate learners’ thinking. Thesession began highlighting. TheGambia’s current education systemand the curriculum materials, whichreflected 21CS to a certain degree.The results of the OAA Africa ministudy, in which The Gambiaparticipated, illustrated that the skillsmight have been used in classroomsbut only unconsciously rather thanexplicitly. To prime the teachers,they were asked to list the 21CS thatthey were aware of and state howthey assess those skills in theclassroom. Although skills such asproblem solving, critical thinking, andeffective communication wereidentified, the teachers wereuncertain about how these skillscould be assessed in theirclassrooms. One of the majordiscussions was aroundunderstanding what skills mean andhow they are different fromknowledge. Despite recognition of theimportance of 21CS, teachers feltthat there were real challenges toteaching these skills in theclassroom. These included:

Two target skills, problem solving andcollaboration, were selected for thepurposes of the OAA project to serve asconcrete examples to illustrate the taskdevelopment process. Both skills wereexplicitly mentioned in the threecountries' education policydocumentation. Once the two skills were decided upon,the next step in the process was todefine and deconstruct the skills intotheir components and subcomponents.Taking into account research on thestructures of problem solving andcollaboration, as well as existingframeworks that identify both theprocesses and components of the skills,the National Teams worked togetherthrough numerous iterations to developa framework acceptable and relevant forall three countries. For problem solving,three components were identified:information gathering, planning asolution, and managing information.Within these components, subcomponents were identified, as well.For example, information gatheringincludes both asking questions relatedto the problem and organizinginformation.

Subcomponents ProcessesSkills components

Ask questions related to theproblem (Aq)Organize information (Oi)

Classify (Cla)Analyze (verify, discriminate, compare (Ana)Describe (Des)

Information gathering (IG)

Generate ideas, options,hypotheses (Ge)

Hypothesize (Hyp)Consider and compare options (ConCom)

Develop plan (Dp) Discriminate (Dis)Identify relationships (Rel)Predict (Pre)

Planning a solution (PS)

Follow a plan (Fp)Managing information (MI)

Compare outcomes with plan (Cf) Compare evidence with predict (Com)Check logical flow (Clf)

Justify the process (Ju) Explain (Exp)

Synthesize (Sy) Summarize (Sum)


THE TARGET SKILLS: PROBLEM SOLVING ANDCOLLABORATION

These subcomponents were furtherdeconstructed to identify more specificprocesses, such as classifying,analyzing, and describing. Forcollaboration, four components wereidentified: participation, communication,negotiation, and decisionmaking. Similarto problem solving, subcomponents werealso identified. Tables 1 and 2 show theframeworks for problem solving andcollaboration, respectively. These twoframeworks set the foundation for thedesign, development, and pilot ofclassroom-based assessments of 21CS.

Table 1. Framework for problem solving

PAGE 12

Lazarous Kalirani Kays and Beatrice Mbewe sharing theirknowledge at Zambia's Stakeholders' Orientation Workshop inOctober 2019

The Gambia: Understandingproblem solving and collaboration Describing problem solvingas a process, rather than a type oftask, helped teachers to betterunderstand the skill itself. Discussionsaround collaboration also took placeto help teachers understand theconcept and distinguish it from skillssuch as cooperation andteamwork. “While cooperation means workingwith people and sharing ideas andresources, collaboration meansworking with people toward theattainment of a shared goal.Collaboration involves workingtogether as a group, assigning roles,and supporting one another towardthe successful accomplishment of thetask. This means everyone takesresponsibility and contributespositively toward the success of thelarger group by effectivelycommunicating, actively listening,taking turns, negotiating, andcompromising,” Mr. OusmaneSenghor, Head of Assessment Unit,MoBSE

Skills components

Subcomponents

Participation (P) Take responsibility (Tr)Share (Sh)Take turns (Tt)Engagement (En)

Communication (C) Receptive (Re)Expressive (Ex)

Negotiation (N) Compromise (Co)Perspective taking (Pt)

Decision making (D) Analysis (An)Evaluation (Ev)Plan (Pl)


PAGE 13

Table 2. Framework for collaboration

"Skills are teachable, andto be acquainted with

21CS, one needs tounderstand the skills andsubskills involved in each

of the processes…Problem solving is not

only limited tomathematics but can alsobe used in other subjects.

A problem arises when oneis faced with a situationthat has no solution andrequires some rigorous

processes —ranging frominformation gathering,

analysis, development of asolution, evaluation ofoptions, and decisionmaking. A problem is

simply a complex situationthat requires a solution.”

Mr. Momodou Jeng, Director, Science andTechnology Education and of In-service

Training Unit, Ministry of Basic &Secondary Education (MoBSE)

RE-VISIONING EXISTINGITEMS TO TARGET 21CSRather than starting from scratch, OAAAfrica relied on existing items and tasksfrom national and classroom levels thathad been identified through the earlierAfrica mini-study (UNESCO, 2020) ashaving the potential to target 21CS.These existing items and tasks wereused as starting points but re-visionedor modified to more clearly target theskills of interest.

Imagine you are outside your houseplaying with your friends. Your parentcomes home and tells you to go andclean up your room and arrange yourtoys. You don’t want to stop playing.You know your room is messy. Whatwould you do?

Can capture the skills and theircomponents; Can capture these skills at increasinglevels of proficiency; Are recognizable to teachers ascapturing the skills; andHave structural features that can bereplicated by teachers.

The goal was to develop items and tasksthat:

To illustrate this process, countriesidentified existing tasks with thepotential to capture 21CS. For example:

The first issue to consider waswhether this item could captureproblem solving. Using theproblem solving framework (Table1), the teams considered how theitem could be modified orexpanded, so that the main skills'components and subcomponentscould be more explicitly captured.Several questions were addedwith the identification of thecomponents and subcomponentsthat were being targeted: 1. What is the problem you arefacing?[Information gathering - read theinformation, gather the relevantpieces, and organize theinformation.]2. What additional information doyou need before answeringthe question?[Information gathering - askquestions related to the problemand consider what informationmight be missing.]


3. What would you do in thiscase? Name 3 possible solutions.[Planning a solution - generatedifferent ideas and options for howto respond.]4. Of all possible solutions, whatis the best and why?[Planning a solution - considerand compare the different possiblesolutions, in order to identify thebest solution and explain why.]5. How will you do this? List thesteps you would take to implementyour solution.[Planning a solution - afteridentifying the best solution,develop a step by step plan forhow the solution will beimplemented.]6. If this solution does not work,what else can you do?[Managing information /Planning a solution - comparethe solution with the plan, checkthe logical flow of their plan, andas necessary draft a newsolution.]

Another approach to re-visioning tasksis shown in Figure 2.

Figure 2. Example item that was expandedto target problem solving

Q1. Which combination of pots can beused to measure 550 ml? A. 400 and 500 B. 150 and 400 C. 750 and 1000 D. 150 alone

Q2. If you want to distribute 2200 mlof water evenly to four friends, explainhow you would do this.

PAGE 14

Items’ capacity to capture theintended skills, components andsubcomponents The targeting of the items to theability levels of the studentsUsability of the items:

Did the students havedifficulty understanding theinstructions? Did the student have everythingthey needed to respond (e.g., pen,paper, space)?

Other issues related to item checking: Did students provide evidence ofpossible misconceptions? Did something unexpected occur? Did the students express interestor frustration?

During a think aloud session, studentsorally report what they are thinking asthey complete the task, providingvaluable information about the internal,and invisible, processes that are beingprompted by the task. These sessionscan provide teachers and taskdevelopers with information about howstudents approach a task and anyfunctional issues that need to beaddressed in revising the task (Leighton,2017). See Appendix B for think aloudguidelines. During the think alouds, teachers orobservers are asked to reflect on thefollowing:


The original (Q1) item is retained as astraightforward numeracy item, with asecond (Q2) item added to targetproblem solving, and drawing on thecomponents and subcomponents ofInformation Gathering > OrganizingInformation, and Planning a Solution >Generating Ideas, Options, andHypotheses. Countries initially engaged in thisprocess of re-visioning existing items,and then generalized the process. Thegeneralization enabled the developmentof several models, or templates, thatcould be used to structure newassessment tasks for problem solvingand collaboration. The templates rangedfrom selected response items (e.g.,matching and multiple choice), toconstructed response items (e.g., fill inthe blank and short answer), and toperformance tasks. Exemplar tasks andassociated templates are included inAppendix C.

Think aloud sessions

Although re-visioned items wouldtypically be based on existing items,there is a need to verify whether thetargeted skills are actually elicited by there-visioned or expanded items. It may beclear, for example, that Q1 in Figure 2targets numeracy as an establishedconstruct. However, the extension of thetask to a new construct—problem solving—needs to be checked. Therefore, theOAA Africa National Teams conductedthink aloud sessions for such items.Think alouds, or what is termed in theacademic literature as cognitivelaboratories, is a method of studying thecognitive or social processes calledupon by tasks as students work throughthem (Griffin & Care, 2014).


PAGE 16

For the teachers to learn about thinkalouds as a process that allowsstudents to explore their thoughts indealing with a task.For teachers to be acquainted withthe procedure and manner in whichthink alouds are administered tolearners.For teachers to begin thinking abouthow to develop similar test items andadminister them to their studentsappropriately.

The Gambia: Think aloud To support teachers in conducting thinkaloud sessions with their students, theNational Technical Teams in eachcountry held a training session forteachers. For The Gambia, the goals ofthe training were:

It was emphasized to the teachers thatthe focus should not be on whether thestudent was able to answer the itemcorrectly, but rather, on the processesthat led the student to the answer. Forexample, when the student is facedwith an issue, how does the learnerthink? What are the processes he/sheconsiders and the skills, components,and subcomponents he/she has appliedto arrive at the answer? Understandingthe answers to these questions goeswell beyond whether the response isright or wrong, which has typically beenthe focus of our teachers. After the discussion, teachers wereplaced in groups and instructed toadminister the think alouds to othergroup members to practice what theylearned. They were tasked withidentifying a group leader, secretary,teacher, and a student, and to reporttheir observations. Once the session was over, teacherswere asked to administer the items tostudents in their respective schools andprovide feedback. The teachers foundthe think aloud sessions with theirstudents both challenging andinteresting.

They stated that their students hadnever been exposed to such kinds oftasks, thus making it morechallenging for the students. At thesame time, students really enjoyedtaking part in the think aloudsbecause it was different. Theteachers also began recognizing theskills related to problem solving andcollaboration and noticing specificbehaviors when students wereworking on the task. For example, inone collaborative task, studentsdiscussed the question amongthemselves to develop a singleresponse where they could agree.During the course of the discussion,they debated and countered eachother’s opinions before coming to aconsensus. However, one of theissues that emerged was thatbecause collaborative tasks werenew, they had not been exposed tohow to structure the collaborativework. At times, students found itdifficult to understand what they weresupposed to do as part of the task,especially if the items were notmultiple choice or in a format withwhich they were familiar. As such, insome groups, one person tended todominate the whole session, whileothers just observed; in other cases,one or two students would dismissothers’ ideas without consideration,thereby not engaging in collaborativeprocesses. When the teachers sawthese different levels of competence,the tasks’ capacity to captureindications of different studentperformance became clear. During the think alouds, teachersmade observations and providedfeedback about specific items, suchas which were too difficult tounderstand due to language issues,or which were not appropriate interms of the content. Quite apart fromthe utility of the think aloud methodas a process within taskdevelopment, the teachers becamemore aware of the variety of studentresponses to curricular content.

each successive description implies ahigher level of performance quality;behaviors are directly observable; inferences can be made aboutdevelopmental learning—there shouldbe no counts of things right and wrong;there is no differential weighting ofresponses - allow the rubrics to accountfor differences in performance;just one central idea is recognizedthrough the increasing levels of quality;comparativeterms such as more or better, are notused to define quality; transparent language is used (nojargon).

Characteristics of high-quality itemsinclude a clear intention; languageunderstood by most students; a simple andauthentic context; and strong probability ofachieving acceptable answers to thetargeted skill. To develop items that cantarget the skills, creating stimulus materialwarrants careful attention. A good stimulusis rich and interesting; is optimallychallenging (i.e., not too difficult or tooeasy); does not pose artificial challenges;offers opportunity to pose searchingquestions; offers opportunity for studentsto show what they know; and is equallyaccessible and equitable for students ofdifferent abilities. How teachers interpretand record student performance is equallyimportant to consider when developingitems. Therefore, to minimize the influenceof variation in interpretation andsubjectivity, development of a set ofscoring criteria or “rubrics” contributes toconsistent marking. A rubric can bedeveloped by setting precise guidelines forjudging students’ work. There are several recommendations forwriting rubrics which include descriptionsof performance across levels of quality.Ensure that:

The data from all think alouds acrossthe three countries were consolidated.Then, based on the data and feedbackfrom teachers and observers, the taskswere revised, along with the templateform for each which describes eachitem structure. Use of these templatesis intended to make it easier to createnew tasks that can elicit the same skillsacross different subject areas andgrade levels. The templates basicallyact as a guide for task and itemdevelopment. Additional guidancedescribing the rationale anddevelopment can be found in the fourthreport of this OAA series.

What are the characteristics of high-quality items?What makes a good stimulus for anitem?How do you design a good scoringrubric?

Before generating new items, eachNational Team decided which subjectareas and grade levels to focus on forassessment development (Table 3).Within the subject areas, specific topicswere selected as starting points. The intent was to develop items that: 1) capture the skills and thecomponents of interest; 2) capturethese skills at increasing levels ofproficiency; 3) are recognizable toteachers as capturing the skills; and 4)have structural features that can bereplicated by teachers. Keeping thesein mind, three main questions werediscussed for generating new items:


GENERATING NEW 21CSITEMS BASED ONTEMPLATES

Table 3. Grades and subject areas for each country

PAGE 17

6 and 8Zambia Social studies, Math and science

Country SubjectsGrade Level(s)

6DRC Math, health/environmental/science, and technology

Gambia Social and environmental studies, English language, math, and science4,5, and 6

Building on the previous workshops on21CS and think alouds, the NationalTeams held additional sessions in theircountries to engage teachers in theprocess of item development. Teamsworked across two different approachesto item development (Figure 3). Through both approaches, NationalTeams supported teachers to try theitems in their classrooms, collectfeedback, and refine the items based onthe feedback. These processes identifiedand addressed challenges and issuesthat arose and helped teachersunderstand the implications of the waythey developed the items. Each National Team worked withteachers in their respective countries todevelop eight tasks, for a total of 24tasks across the three countries, thattargeted problem solving and/orcollaboration in the subject areas ofenvironment, mathematics, health,English, science, and social studies.There was a mix of task and item types,including dichotomous (correct/incorrect)response, closed constructed response,open constructed response, andperformance.

Once the tasks were developed, theNational Teams met together with theirteacher teams to panel the tasks andtheir associated items. The panelingprocesses were undertaken in eachcountry in slightly different ways but allwith the same goal. Paneling is aprocess to check and improve items forthe purposes of quality assurance,establish content and construct validity,and explore inadequacies in items, andreduce waste in piloting of inadequateitems.


PAGE 18

TASK AND ITEM DEVELOPMENT APPROACHES2

Figure 3. Two approaches for item development

Paneling of tasks: National andcollaborative

Team members from The Gambia, and Zambia paneling tasks

Approach 1 Approach 2

Teachers identify existing assessmentitems that they have used in the past with

their students

Teachers "tweak" or revise the existingitems using the task templates to guide

them

Teachers identify a topic or subject areafrom the curriculum

Teachers use the task templates todevelop new items from the topic/subject

area they have selected

whether the item assesses (part of)the construct ;what students need to know toanswer the question and whether the curriculum has coveredthis ;the authenticity of the item ;the precision and clarity of the itemand its phrasing ;the amount of time needed toproduce an answer ;adequacy of the scoring rubrics ; and equity for students ofdifferent backgrounds.

Ideally each paneling group wouldinclude two or three independentexperts, a representative of the itemwriters, and a couple of teachers at thetarget grade level and subject area usedas the base for the tasks. Thisdistribution would ensure the expertiseto examine and revise test items isavailable. A chairperson or group leader shouldfacilitate and manage the discussion ,and summarize what needs to be doneto revise the items . The independentgroup members and teachers shouldreview the tasks individually and thenshare perspectives, before requestingclarification or inputs from the itemwriters. The role of the latter is torespond to these requests rather thandefend their items, and explain therationale for the various features ofthe items. At the end of the process, adecision is reached concerning whetherto discard, amend, or approve the tasksand accompanying items. In addition,the comments and rationale fordecisions are noted to ensure the teamshave all of the necessary information. The panel assesses items based on thefollowing criteria:


PAGE 19

Following the country-based panelingactivities, both the newly developed,as well as previous tasks, were againpaneled and reviewed in the fourthOAA workshop through a collaborativeprocess. Beyond the review of items,the OAA workshop’s focus was on thescoring criteria for items. Review ofscoring criteria acts as anotherstimulus and opportunity to analyzethe tasks and items themselves, sinceit is the specificity of the scoringprotocols and coding that tend tohighlight previously missed issues. Asignificant input to the review wasdocumentation of student responsesto the items from the think aloudactivities in each country. Throughinterrogation of the written responsesof the students, the full richness oftheir varied ways of thinking shedslight on the strengths and weaknessesof task and item design. In thecollaborative workshop, sets of 10-15student responses to each item werereviewed, categorized at differentlevels of quality, and then referredback to the scoring criteria. On thebasis of the process, the scoringcriteria and rubrics were reviewed,and in some cases, the actual itemsthemselves were amended.

DRC, The Gambia and Zambia team members workingacross languages

PILOTING OF TASKS3The purpose of the pilot activity was tocheck whether tasks such as those thathad been developed could beadministered at a classroom level, andwhether student responses wereinterpretable in terms of thehypothesized skills. Analysis of theresults would enable further finessing ofthe task templates and provide moreinformation about likely studentcapabilities in these previously untested21CS to guide future teaching andassessment. Based on feedback from thecollaborative paneling sessions, seventasks from across DRC, The Gambia,and Zambia were dropped. The rest ofthe tasks were adapted across countriesas needed for the pilot. Three tasks fromthe Asia pilot (see “OAA: Focus on Asia”report) were also included, against thepossibility that future studies might linkthe data from the Asia and Africa pilots.The pilot required students in eachcountry to complete the tasks inclassroom conditions. Grade 6 studentsin DRC, Grade 7 students in TheGambia, and Grade 6 and 8 students inZambia participated in the pilots. Table 4shows the pilot tasks administered ineach country.

Selection of schools; Slight revisions and translations oftasks as needed;Training sessions around testadministration and scoring withteachers, data collectors, and otherstakeholders participating in theproject;Test administration;Scoring processes; andData entry.

The piloting process included severalsteps at the country level:

Each country approached the pilotprocess slightly differently. To provide acountry perspective, DRC describes itsprocess.


PAGE 20

Table 4. Pilot items

Note: Tasks are identified by the country which developed them: Z = Zambia items; G = The Gambia items; D = DRCitems; and A = Asia items. G3 and G4 items by The Gambia were re-labeled for the pilot.

Classroom tasks act as stimulus for assessment revis

The DRC: Pilot of the tasks The National Team from DRCdeveloped a timeline for the pilot,planning to start the various activitiesmid-September, two weeks after thestart of the 2019-2020 school year.However, the timeline had to be shifteddue to unforeseen circumstances,including delays in receiving the fundsto implement the pilot; the mandate forexperts to implement the free basiceducation announced by PresidentFélix Tshisekedi; and the preparation ofthe Mid-Term Review of the Project forthe Improvement of the Quality ofEducation (PAQUE) funded by theGlobal Partnership for Education. Thus,the piloting activities started in mid-October, rather than earlier. For the DRC, the items were pilotedwith Grade 6 students, as the itemswere appropriate within the context oftheir curriculum. The items were basedin mathematics, science, environment,and health, with five items for problemsolving and two for collaboration. Fromthe three items written by Asiancountries, the DRC team and the itemwriters chose one. Training sessionsTo prepare for the training sessions,the National Team members developedmaterials and guides, including apresentation of the overall study, theconcept of 21CS, presentation of theitems to be included in the pilot, andscoring processes, as well as logisticsassociated with students in physicalclassrooms for test administration. Twotraining sessions were conducted intwo different areas. In the city ofKinshasa, the capacity buildingworkshop took place on October 10,2019. It involved 11 teachers and foursupervisors (the Deputy InspectorGeneral in charge of Assessments,Director of School Guidance, and twoitem writers).


PAGE 21

In Mbanza-Ngungu, the capacitybuilding workshop was organized onOctober 18, 2019 with 15 participants,including the Provincial Director ofEnseignement Primaire, Secondaire etTechnique (EPST), the PrincipalInspector of EPST, the Sub-Proved, theSenior Provincial Inspector in charge ofEvaluations, Teaching Advisers, andthree teachers. The training sessionscovered how to administer the tasksand how to score student responses. Selection of schoolsTwo schools selected at the beginningwere replaced. A school in Kinshasawas replaced due to construction on theroad leading to the school; a ruralschool in Mbanza-Ngungu was replacedbecause there was heavy rain, andschools were cancelled the day beforethe pilot was scheduled. Thus, inKinshasa, Primary School (EP) Kengo,urban private school was replaced byEP1 Manyanga; and in Mbanza-Ngungu, EP ZANZA by EP1 KOLA. In the end, five schools were selectedin the city of Kinshasa and three othersin the Kongo Central Province, aroundthe city of Mbanza-Ngungu, 150 kmfrom Kinshasa. The selection of schoolswas made with the assistance of theProvincial Directors of EPST, aftercommunicating to them the desire tohave a range of schools that varied onlocation, size, and socio-economicstatus. Based on the criteria, theProvincial Directors consulted theHeads of Education Divisions andSchool Directors. Task administrationThe administration of the tasks involved120 students, including 75 from the fiveschools in the city of Kinshasa and 45from the three schools in KongoCentral. With the delay in starting theactivities, the team chose to group thestudents in one school in Kinshasa andone school in Mbanza-Ngungu.

The items were given to students atthe beginning of Grade 6; however,the items were developed based onthe full Grade 6 curriculum, whichmeant that the students had not yetcovered much of the content uponwhich the problem solving andcollaboration skills were to beapplied.The vocabulary appeared too difficultfor many of the students tounderstand, and needed reviewprior to further use.The teachers understood theimportance of the project and wantedto use these types of assessmenttasks to encourage students todeeper reflect.Teachers mentioned the need forbetter training for themselves todevelop and use assessments of21CS.

The Kinshasa site was at BOBOTOCollege, which received 60 studentsfrom four schools. To note, having alarge group of 60 students from fourKinshasa schools in the same roommade student supervision difficult,especially with the lack of trainedteachers. The Collège Des Savoirsopted not to send their students andinstead held the pilot within their ownschool. In Mbanza-Ngungu, the childrenwere grouped together at EP1 KOLA. The administration of the itemsrevealed one of the characteristics ofthe level of teaching and learning in theDRC, namely performance disparitybetween urban and rural areas andbetween schools in the sameenvironment. Indeed, the students ofMbanza-Ngungu all claimed to havereading and writing difficulties. Theymainly answered items that weremultiple choice and had great difficultywith items that required writing. The following are some observationsbased on the administration:


PAGE 22

Scoring processNotwithstanding the training that hadtaken place, and teachers' practice ofthe scoring process, it became clearthat there had still not been sufficienttime for adequate training in scoring.Thus, there were some issues aroundscoring, especially aroundunderstanding which responses alignedwith which level of quality described inthe rubrics. Data entryThe data entry was undertaken over avery brief period, and as such it wasdifficult for the national team to ensurequality control across actual test formsand the database. The process wascarried out in accordance with theinformation provided in the codebookas closely as possible.

KEY POINTSAlthough each country followed similarprocedures, they each made differentdecisions on some key points. Forexample, unlike DRC, The Gambiaadministered items in their pilot torecently graduated Grade 7 students.This decision was made in the light ofthe point made by DRC, that students atthe targeted Grade 6 level would not, atthat time of the school year, have yetcovered all the curricular content thatwas assumed by the items. The decisionby Zambia to include both Grade 6 and 8students provided them with a greateropportunity to explore the impact ofcurricular knowledge on the applicationof skills. The issue of how much training isnecessary for pilot implementation isalso raised by the slightly differingapproaches of the countries. As statedby DRC, factors outside the control ofthe education team impacted the amountof training provided to participatingpersonnel.

In the meantime, in The Gambia, trainingwas provided over a three-day period,touching on the genesis and backgroundof 21CS, the specific skill areas ofproblem solving and collaboration, andthe practical guidelines for administrationof the tasks and scoring protocols. ForZambia, awareness of the large classsizes (averaging 80 learners per class)was factored into the training. Inparticular, the collaboration tasks poseda challenge for administration, withteachers needing to ensure systematicsampling to select the learners in eachclass. In each school, 42 students madeup 14 groups of three learners. Thismade it possible for the trained teachersto assess the three-student group tasks. These differences highlight thesignificance of the need for systemicaction on professional development forteachers as countries update curriculargoals. Shifting from a knowledge base toa competency-base requires bothpedagogical and philosophicalcontextualization. These issues arediscussed in the fifth report of this series.


PAGE 23

Collaborative tasks in Zambia, and problem solving in nutritionclasses

WHAT DO THEPILOT DATA TELLUS?

4

Identify how the tasks and their itemsfunction;Determine limitations of the tasks anditems and provide suggestions forimprovement; andProvide user-friendly item maps todemonstrate targeting—that is, howwell the tasks and items are matchedto student ability.

The data from the pilot were analyzedseparately for each country due to therelatively few tasks common across thecountries. The objectives of the dataanalysis were to:

The information from the pilot data thenprovides the countries with feedback onfunctionalities of the tasks. The taskadministration component also providesinvaluable information about studentresponses as they struggle tounderstand these different approachesto assessment. And the scoringprocesses provide scorers and teacherswith additional insights into how theirstudents demonstrate the proficienciesof interest. Most importantly, the overallpilot provides evidence to support thevalidity of the assessment approach andthe usability of the templates for futuretask and item development.

THE PILOT DETAILStudent responses to the assessmenttasks were derived from the studentsamples in Table 5. Note the differencesin numbers of male and female studentsfor DRC and Zambia. Figure 4 showsdifferences in the ages of theparticipating students across the threecountries. The three countries drew onstudents across age ranges and grades,and is illustrative of the flexibility of theproject in achieving its goal—to exploreassessment approaches and specifictypes of tasks—as opposed to beinginterested in comparative studentperformance across countries.

The 19 tasks and their constituent itemswere distributed across problem solvingand collaboration skills, with some baseitems in each reflecting content in therelevant subject area alone. Such baseitems, where they occur, are typicallythe first items within a task, andestablish the context for the applicationof the 21CS. The main components andsubcomponents of the two skills that arerepresented across the administeredproblem solving and collaboration tasksare shown in Table 6. Although tasksand their items were deliberately draftedto target the skills, there was not a morestructured plan in which specificnumbers of items were to be written tospecific components. Hence, thedistribution of items across the variouscomponents provides a quick image ofwhich components are possibly the mosteasily targeted using the processesadopted.

Collaborative subcomponents that arenot captured by the tasks include turntaking, engagement, receptivecommunication, evaluation, andplanning. Problem solvingsubcomponents that are not captured bythe tasks include following a plan,comparing outcomes with plans, andsynthesis. Some others are alsoinfrequently captured directly, althoughthey may be embedded within otheractions. For example, although there isno one item that captures “follow a plan”directly, it is clear that the student needsto follow a plan in order to “consider andcompare options”; similarly there is noone item that captures just “to askquestions related to the problem”,although the student must have donethis mentally to organize or classifyinformation. A process such as“analysis” also clearly underpins manycognitive operations.

The reliability of the problem solving andcollaboration scales were calculated foreach country dataset. The (EAP andWLE) coefficients for The Gambia wereall above 0.80, demonstratingacceptable levels. Reliability for scalesfor DRC and Zambia did not reachacceptable levels based on the cut-offsapplied. The reliability analyses must beinterpreted with great caution for severalreasons. First, the numbers of studentsare not large. Second, some difficultiesin coding and scoring were experiencedin the countries. And third, since sometasks included aspects of both problemsolving and collaboration, analyzingthese together makes theunidimensional assumption that theconstruct of “collaborative problemsolving,” as distinct from two separateconstructs of problem solving andcollaboration, is being analyzed. Theconcept of collaborative problem solvingis contested (Care & Griffin, 2017).


PAGE 24

Figure 4. Age distribution of studentsacross the three countries

DESCRIPTION OF THE TASKS

Table 5. Total number of students in the pilotby gender across three countries

ANALYSIS

Country

DRC

The Gambia

Zambia grade 6 and 8

Males

73

103

156

Females

73

103

156

Total

120

209

348

Consequently, for optimal use of pilotresults, attention must be focused oninterpretation within a task. Analysis ofindividual assessment tasks and theiritems generated information useful forrectification of some of the scoringrubrics, as well as recommendations foramendment of task main stimulus, orstem, of the tasks, and their items. Themain issues encountered at the itemlevel were: (1) lack of use of somescoring categories; and (2) lack ofassociation of some scoring categorieswith overall performance (as indicatedby zero-level point biserial coefficientsfor non-zero scoring categories, ornegative coefficients associated withnon-zero scoring categories). Furtheranalysis is required within countries toverify whether low frequencies of somescoring categories are due to non-appearance of the responses that wouldpopulate those codes, or codingdifficulties experienced by the scorers.The concerns expressed by the DRCnational team about adequacy of scoringtraining may be germane to this issue. One of the difficulties associated withassessment of complex skillsets is theaccuracy with which particularresponses or actions can be associateduniquely, or primarily, with a particularskill or subskill. Some of the items couldbe seen as indicative of different,although complementary, competencies.

Component ProcessesSubcomponent

Problem solvingInformation gathering Classify

Analyze Describe

Organize information14 14 328

Planning a solution Hypothesize Consider and compare

Generate ideas and options19 15 79

Managing information ExplainJustify the process5 4 8

CollaborationParticipation Share10 10

Communication Express7 7

Negotiation Compromize7 6

Decisionmaking 8


PAGE 25

Table 6. Frequency of skill components and subcomponents represented in the pilot tasks

Note. Numbers of subcomponents and process activities do not necessarily sum to totals in the components column, since some of theformer are superordinate to the components.

This reality could influence the difficultyof coding, particularly for educators whohave been familiar with assessment thatis focused on correct versus incorrectresponses to knowledge-based items.The collaboration tasks and their itemsrequired greater rectification than did theproblem solving tasks and items. Thiscould well be due to educators' lesserfamiliarity with collaboration as acompetency. All tasks included multiple items. Thisapproach was taken to make test-takingtime efficient. Where just one set ofstimulus materials can generate multipleitems, the overall reading load forstudents is minimized. Item difficultytherefore depends not only on the initialstimulus, but also on the characteristicsof each item. One assessment task cantherefore include both relatively easy aswell as average and difficult items. Oneof the advantages of this style ofassessment is that examination of theitem difficulties can be used to makedecisions about best sets of items to usewith students at different competency orgrade levels, as well as scoring rubricswhich further differentiate ability withinitems. As an example, Figure 5 showsthe item-person map for DRC's studentresponses to problem solving tasks. Themap includes six tasks (D1, D2, D3, D4,D8, A1), with their items and varyingdifficulty levels of responses to these.

The six tasks were designed to assessproblem solving across a variety ofscience, social sciences, andmathematics topics. Figure 5 depictswhat is more and less difficult forstudents, with items that appear towardthe top of the graphic being moredifficult than those toward the bottom.(For a description of how to interpretsuch graphics, please see the "OAA:Focus on Asia" report.) As an example, Task D1 (a task createdby the DRC team) is shown withresponses ranging from low level ofdifficulty at a logit level of about -3.0with item D1a_Cat1 to D1a_Cat3 atabout logit 2.8. (“Cat” or category hererefers to the quality level of theresponse, with Cat1 denoting the mostbasic quality level.) The task itself dealswith issues associated with poliomyelitis,which is a current challenge for the DRChealth system and society (World HealthOrganization, 2020).

The easiest item for D1 requires astudent merely to attempt the questionposed; the next level of difficulty isachieved when the student can identifyeither a cause or risk of poliomyelitisreflecting subcomponents of problemsolving such as information gathering,organizing, and describing (D1a_Cat2),and then identifying the associationsbetween a cause or effect (D1a_Cat3). Ahigher level is in principle possible—thatof identifying two causes with theirassociated risks—but no student in thissample provided this level of response.For item D1b, the student is required togenerate hypotheses within the context ofthe problem scenario. As can be seen,three levels of quality response areprovided at D1b_Cat1, Cat2, and Cat3depicting increases in difficulty fromproviding irrelevant responses toidentification of actions, and then linkingthese with the target outcome throughgeneration of hypotheses. It is salient to note that the three OAAAfrica countries wanted to acknowledgestudents' effort. This translated intoscoring protocols in which an irrelevantresponse was treated as a higher qualitythan no response.


PAGE 26

Figure 5. Problem solving distribution of DRC students against tasks

Key: Example D1a_Cat1 = DRC task number 1, item a, response at category level 1


PAGE 27

TASK REVIEWThe value of pilot results such as theselies in the interpretation of the data, inexamining what sort of tasks, items, andresponse categories are easier or moredifficult. Knowing what is challenging forstudents helps the teachers (and ofcourse the developers of assessments)identify what their students are ready tolearn next, what might need to beconsolidated, and what might becurrently beyond the learning readinessof their students. Based on review of the tasks and theiritems, there were no tasksrecommended for deletion, or in otherwords, deemed useless. Just four itemswere recommended for deletion.Amendments to scoring criteria wererecommended for many items, with asizable number constituting amendmentor deletion of a scoring category, ratherthan amendment of the item itself. Thisbrings into question whether the scoringcategories, the scoring processes, orboth are the source of some anomalies.Happily, where the same tasks wereadministered across more than onecountry, the same anomalies were found—indicating that the items themselvesrather than the samples need to beexplored. Scoring categories recommended fordeletion included those for which therewere very few correct responses—indicating that the items were toodifficult for the students.

The DRC Team in the final workshop reviewing the scoringcriteria

In other cases, the association betweenthe item response and student scorewas weak, indicating that the item wasnot functioning in the expected way.Where the mean ability did not increaseacross increasingly difficult responsecategories, items were also consideredunsatisfactory. Several of these factorssometimes combined to lead torecommendation for deletion of responsecategories. For example, the first item in one taskasked students to identify twoadvantages and two disadvantages ofthe rainy and dry seasons. The scoringcategories allowed for no response,irrelevant response, one advantage ordisadvantage, and two advantages anddisadvantages. Although the itemcategories functioned appropriately forthe first three categories, this was notthe case for the final one. In retrospect,this was due to a rubric problem in that itdid not follow a logical sequence andwas managed differently by countries. Atthe same time, the low frequencies forthe highest level responses wereprobably accounted for by theexpressive literacy levels of students.The DRC OAA team noted that studentswere less well prepared to cope withquestions that required a lot of writing,as opposed to responding to multiplechoice or short answer questions. TheZambia OAA team commented post-pilotthat there were reading challenges forsome of their students. In this same task for the second item,and its three sub-items, as it shifted toassessment of collaborationcomponents, students were required toagree on the most important advantageand disadvantage of the two seasons,based on their pooled earlier responses.Again the “easiest” levels of responsesfunctioned appropriately, but responsesthat demonstrated students' negotiationwith each other through linking of ideaswere very few.


PAGE 28

The Zambia OAA team commentedthat their learners appeared to havefew group discussion skills, wereunused to conventions of sharinginformation and then providingfeedback, and were shy. In addition,the concept of collaborating asopposed to competing led somestudents to be reluctant in theirsharing of information and ideas. Analysis of the pilot results allow notonly for critical review of the tasksand items themselves, but also serveto highlight cultural learningconventions that have been nurturedin schools for many years. Moving toassessment tasks that value studentthinking, discussion, perspective, andinterpretation is a philosophical shift—for students, as well as in somecases, for teachers. The pilot experience provided thenational teams with a rich source ofinformation to draw on for theircontinuing 21CS task development.The training, task administration, andscoring issues provided an equallyvaluable resource to inform the threeeducation systems about theinfrastructure required to effectchange.

CONCLUSIONThe approach adopted by OAA Africafocus countries is symptomatic of abottom-up approach to change ineducational assessment. It takes theposition that assessment forms andpractices need to be understood by thoseeducation practitioners most closelyinvolved in the actual administration andinstructional use of assessment data.Beyond just understanding, the approachtakes the position that practitioners needto create and develop their ownassessments, and that these assessmentsshould be aligned with technical expertiseat central education department orministry levels. Rather than adopting amodel in which teachers and schools aremere receivers of assessment materials, itis a model where assessment becomes apart of instructional design. Adoption of such a model carries with itrisk, of course. The major challenge isensuring that teachers and schoolleadership teams receive the professionaldevelopment inputs that are needed forimplementation. Even in the small pilotstudy reported across the three countries,it is clear that the professionaldevelopment delivery varied considerablyacross countries. This is in no way due tofault or inadequacy on the part of thenational teams, but is a function of thevaried resources and infrastructure ineach country, as well as the realities ofpolitical and economic issues, healthcrises, and conflicts.

Long-term engagement in problem solving and collaboration leading to common understandings and friendships – au revoir

Care, E. & Griffin, P. (2017). Assessment of collaborative problem-solving processes. InB. Csapó, & J. Funke (Eds.), The nature of problem solving. Using research toinspire 21st centurylearning. (pp. 227-243). Paris: OECD Publishing.

Care, E., Kim, H., Vista, A., & Anderson, K. (2018). Education System Alignment for

21st Century Skills: Focus on Assessment. Center for Universal Education at TheBrookings Institution.

Education, E. (2013). The Zambia Education Curriculum Framework 2013. Griffin, P., & Care, E. (Eds.). (2014). Assessment and teaching of 21st century

skills: Methods and approach. Springer. Herman, J. L. (2010). Coherence: Key to next generation assessment success.

Assessment and Accountability Comprehensive Center. Retrieved fromhttps://files.eric.ed.gov/fulltext/ED524221.pdf.

Kim, H., & Care, E. (2020). Assessment in sub-Saharan Africa: capturing 21st century

skills. UNESCO and Brookings Institution. Leighton, J. P. (2017). Using think-aloud interviews and cognitive labs in educational

research. New York, NY: Oxford University Press. Ministère de l’Enseignement Primaire Secondaire et Initiation à la Nouvelle

Citoyenneté, Ministère de l’Enseignement Technique et Professionnel, Ministère del’Enseignement Supérieur et Universitaire, & Ministère des Affaires Sociales, ActionHumanitaire et Solidarité Nationale. (2015). Stratégie sectorielle de l’éducation et dela formation 2016-2025. Available from: http://www.eduquepsp.education/sgc/wp-content/uploads/2018/06/Strategie-sectorielle.pdf.

Ministries of Basic and Secondary Education and Higher Education Research Science

and Technology. (2016). Education Sector Policy 2016-2030 Accessible, equitableand inclusive quality education for sustainable development. Available from:http://www.edugambia.gm/data-area/publications/policy-documents/256-education-policy-2016-2030-web-version.html.

The Ministry of General Education and The Ministry of Higher Education. Education

and skills sector plan 2017-2021. Republic of Zambia. Available from:https://www.moge.gov.zm/download/policies/Education-and-Skills-Sector-Plan-2017-2021.pdf.

World Health Organization. (2020). WHO Health Emergency Dashboard.

(https://extranet.who.int/publicemergency).


PAGE 29

REFERENCES


PAGE 30

APPENDIX A: NATIONAL TEAMS

Country Teams Schools

DemocraticRepublic ofCongo

Zone urbaine Ecole primaire EP1 BOBOTOEcole primaire CS MANYANGA Ecole primaire EPA 2 GOMBEEcole primaire EP1 BINZAEcole primaire COLLEGEDES SAVOIRS (péri-urbain) Zone rurale Ecole primaire EP1 BOKOEcole primaire EP1 KOLAEcole primaire EP MBAMBA

Mr. Jovin Mukadi Tsangala, Conseiller aucabinet du Ministre de ’Enseignement Primaire,Secondaire et Professionnelle, Cabinet duMinistre Mr. Kasang Nduku, Expert chargé de laformation, Secrétariat Permanent d’Appui et deCoordination du Secteur de l’éducation (SPACE) Mr. Smith Mpaka, Coordonnateur de la CelluleIndépendante d’Évaluation des Acquis Scolaires Mr. Mapasi Mbela Chançard, enseignant auCollège des Savoirs Dr. Jerry Kindomba, Country Director, GivingBack to Africa

The Gambia Mr. Momodou Jeng, Director, Science andTechnology Education and In-service TrainingUnit Mr. Ousmane Senghor, Head of AssessmentUnit Mr. Omar Ceesay, Education Officer Mrs. Isatou Ndow, Vice Principal, GambiaCollege Mrs. Saffie Nyass, Deputy Head Teacher

St. Peter's Lower Basic School Mansa Kolley Bojang LowerBasic School Abuko Lower Basic School St. Mary's Lower Basic School

Zambia Mr. Victor S. Mkumba Principal CurriculumSpecialist Social Sciences; Directorate ofStandards and Curriculum Mr. Lazarous B. Y. Kalirani, Principal EducationStandards Officer Tertiary Education;Directorate of Standards and Curriculum Mr. Shadreck Nkoya, Assistant DirectorResearch and Test Development; ExaminationsCouncil of Zambia Ms. Beatrice B. Mbewe, Teacher Vera ChilubaPrimary School; Ministry of General Education

Kabulonga Girls SecondarySchoolMount Makulu Secondary School Parklands Secondary School Vera Chiluba Primary SchoolMatipula Primary School Chibolya Primary School


PAGE 31

APPENDIX B: THINK ALOUD GUIDELINESA “think aloud” activity (also called a “cognitive laboratory”) is a method of studying theskills and subskills that are used by a student when engaging in assessment tasks.Think aloud activities can provide valuable information about whether target skills andsubskills are actually prompted by a task. As students work through the task, they orallyreport their own mental processes (i.e., explain their thinking and reasoning as theycomplete the task) so that these can be recorded by the teacher/observer. These guidelines have been prepared to guide think aloud activities. For individual tasks (i.e., problem solving), a small number students are needed(preferably selected across the range of estimated low, medium, and high ability bytheir teacher). For group tasks (i.e., collaboration), two groups of the required number of students areneeded (ideally including low, medium, and high ability mix of students.) [Note thatdifferent collaboration tasks may require different numbers of students]. The same students can be used for all tasks. However, if the tasks take more than 60minutes, additional students should be selected to participate due to fatigue issues.

INSTRUCTIONS TO TEACHER/OBSERVERTell students to work as independently as possible. Students should "think aloud" asthey complete the task. If students need help while completing the tasks, you should first prompt the student toask his/her group members (if it is a collaborative task), or think some more (if it is anindividual task). If this is unfruitful, you should note the question for which help wasprovided and include a brief description on the think aloud record form. Record the information from each student or group for each task on a single form. Theforms are provided specific to each of the tasks.

READ THIS TO STUDENTS“We are asking you to help us as we create new tasks for students. You will not bemarked on the task–we just want you to help us. So, don’t worry if you do notunderstand anything or are unsure—we need to know how you go about working outwhat to do. The tasks you will be doing are designed to assess [collaboration/problem solving]. Thismeans you will have to work with partners/alone to find out what you have to do andsolve the problems you are given. If you get stuck, you should try and work out whatyou have to do rather than ask me. We asked for your help today because we want to know what you think about when youwork on these tasks. In order to do this, I am going to ask you to THINK ALOUD as youwork on the different problems. What I mean by think aloud is that I want you to tell meEVERYTHING as you work on the task. I would like you to talk aloud CONSTANTLYfrom the time you start, until you finish. I don’t want you to try to plan out what you say.Just act as if you are alone in the room speaking to yourself. It is most important thatyou keep talking so that I know what’s going on.


PAGE 32

RememberSit near the student but not in their personal space.If the student is silent for more than a few seconds, prompt with Keep talking orWhat’s happening?If the student is talking too quietly, say Please speak louder.”Be sure that the student is also providing her/his responses or taking action to thetask, not only talking—say: Please provide/write down your response. If thestudent is having trouble providing responses and talking simultaneously, have thestudent talk first, then provide her/his responses.If the student asks you what to do because s/he does not understand a question, tellher/him: Maybe think about it another way, or if a collaborative task: Ask yourgroup members or do whatever you think makes sense. You should not help themsolve the task.Be attentive with body language by head nodding or smiling in response to students.Do NOT tell the student if s/he got an answer right or wrong.Do NOT tell the student if s/he did well/poorly on the activity.Do NOT show bias for certain questions or item formats (e.g., Do not say anythinglike, “This is not a very good problem.” OR “Problems like these don’t test manyskills.”)

If you stop talking at any point, I’ll remind you to KEEP TALKING. If you feeluncomfortable or don’t wish to continue, you can stop the tasks at any time. Do you understand what I want you to do?”

FacilitationProvide the task. After a couple of minutes of the student having started, give thestudent quick feedback. Tell the student if they need to speak more, or are doing whatyou need. You may need to model thinking aloud in order to help the studentunderstand what to do.

Note takingAs the student completes each task, make notes on the “Think Aloud Record Form.” Many of the tasks may be unfamiliar to the students. We do want to know if there areissues with the way the tasks are presented. You should note these issues in thecomments section of the form.

Usability issues:Did you see evidence that the student had difficulty understanding theinstructionsDid the student have everything needed to respond (e.g., pen, paper, space, orother)?Briefly summarize any usability issues encountered in the comments.

Comments:Summarize any issues not mentioned above. Did students provide evidence ofpossible misconceptions? Did something unexpected occur? Did studentsexpress interest or frustration? Keep a record of when you did not understand what the student was doing.

1.2.


PAGE 33

“Think Aloud” Record Form

Student name

Teacher / observer name

Grade level of student/s

School name

Task number: 1 Circle YES or NO to confirm statements below; otherwiseleave blank. For time needed, please write number of minutes

Target skill: Task appears to test the targeted skill YES NO

Task appears to test the targeted skill YES NO

Task appears to test the targeted subskill/s

Info gather – Organize info – describe YES NO

Plan solution – Generate hypotheses YES NO

Plan solution – Develop plan YES NO

Task was □ easy, □ appropriate, □ difficult for student/s

Students knew how to complete the task withouthelp YES NO

How many minutes are needed for this task? mins

Comment here if there areother kinds of answers thanare currently catered for inthe marking instructions forthis item

Enter observation notes and comments here; includestudent think-aloud commentary and answers.


PAGE 34

APPENDIX C: TEMPLATES AND EXAMPLESEach template presented is followed by a task example. More detail about how suchtasks capture specific components and subcomponents, and their scoring codes, canbe found in the fourth report in this series.

Info gather – Organize info –describePlan solution – GeneratehypothesesTa

sk 1

a) Describe one environmental problem that you know about. What causes the problem, and what is the result?

b) Suggest what can be done to solve the problem.

Tem

plat

e 1 a) Describe a problem (e.g., health, social, science, environmental)

b) Suggest what can be done to solve the problemInfo gather – Organize info –describePlan solution – Generatehypotheses

Tem

plat

e 2

Pose an arithmetic calculation with one number missing and with atleast two mathematical operators, for the student to identify themissing number.Provide at least one inaccurate answer, and ask how this could havebeen calculated.Request that the student create a similar task.

Plan solution – Generatehypotheses – Consider andcomparePlan solution – Develop plan– predict

I think of a number. I multiply it by 3. The answer is 18. a) What is the number? i. 6 ii. 9 iii. 21 iv. 54

b) Your friend Fatima chose answer (iii). How do you think she got her answer? c) Your friend John chose answer (iv). How do you think he got his answer?d) Create a similar numeric task to a), with four response options,

one of which is correct, and two of which resemble the reasoningshown by Fatima and John. (e.g. "I think of a number. I divide it by"x". The answer is "x". etc.)

Task

2

Plan solution – Generatehypotheses – Consider andcomparePlan solution – Develop plan– Predict

Tem

plat

e 3 Item that requires the student to generate [x] hypotheses in order to

explain a situation in terms of cause and effect.

Fertilizer can poison people. Fertilizer was spread in the fields by thefarmers. People in the village became sick due to the fertilizer. How didthis happen? Name one reason if: a) a drought followed straight after the fertilizer was spread; b) heavy rain occurred straight after the fertilizer was spread.

Generate hypothesesManage info – Justify –Explain

Task

3


Seven boys went to play and collected some small stones - shown in thetable. Three students calculated the following:

a) Zane found the mean as 37+25+20+16+23+11+25 = 154Do you agree with Zane? Explain your answer.

b) Davide found the median as the middle data value 37, 25, 20, 16, 23, 11, 25 = 16

Do you agree with Davide? Explain your answer.c) How many more small stones should each collect for the mean to

be 40? Show these in the table.


Task

4Te

mpl

ate

4 Provide information.

a) Show examples of misinterpretation of the information and b) Request explanations for how the misinterpretations were reached.c) What can you do so that these misinterpretations do not recur?

Generate ideas – Consider andcompare Justify – Explain Plan solution – Generatehypotheses

Provide problem that includes a finite amount of a resource. Provide information about different people having different needs for theresource and the amount of the resource needed for each activity. In a group, each student takes on a role and identify:

a) What is best outcome for self? (individually)b) What is the best outcome for all? Each member shares what is

best outcome for self before deciding as a group what is best forall. Explain reasoning (as a group).

Task

5

Generate ideas – Considerand compare Justify – ExplainParticipation – SharingCommunication – ExpressiveDecisionmaking

The mother needs coal to make food.The grandmother needs coal for heat during the day to stay warm andwatch the baby grand-daughter.The young student needs coal for light to study.

7 units of coal to make food8 units of coal for light8 units of coal for heat

Serrekunda is an over-crowded town. Large families includinggrandparents, and their adult children with their wives and husbands andchildren, all live in small two- or three-room houses. One issue thatthese families experience is that there is not enough coal for everyoneto use as needed. Different members of the family have different daily needs and uses forcoal.

Each of these activities require different daily amounts of coal.

The family has only 15 units of coal total for each day. In a group of three students, identify who will take on which family role(mother, grandmother, or young student). Question Set 1:Individually, state what is the best outcome for yourself. Question Set 2: As a group, discuss and come up with the best outcome for all membersof the family. Explain your answer.

Tem

plat

e 5

Generate ideas – Considerand compare Justify – ExplainParticipation – SharingCommunication – ExpressiveDecisionmaking


PAGE 36

Provide a problem or prompt. Then in a group of three students, go through the following process:

Template 6

Student 1 Student 2 Student 3 Subskills

Round 1: Thinkof factors andproposal ofpoints byeach individualmember

Factor andProposal

Factor andProposal

Factor andProposal

Generate ideas – Consider andcompare Participation – SharingParticipation – Turn takingCommunication – Expressive

Round 2:Justification foryour perspectivefrom eachmember

Justificationfor proposal



Justify – ExplainParticipation – SharingParticipation – Turn takingCommunication – Expressive

Round 3:Consensus #1and reason

Agree withMember 2

Own proposal Own proposal Justify – ExplainParticipation– Turn takingCommunication – ExpressiveNegotiation – Compromise

Round 4:Justification

Justificationfor proposal#2



Justify – ExplainNegotiation – Perspective taking

Round 5:Consensus #2and reason

Agree withMember 2

Own proposal Agree withMember 2

Justify – ExplainDecision making

Koffi has breathing problems and coughs a lot. The doctor says his illness is not related to a virusor bacteria, but rather to the air he breathes. Work as a group of three students.

a) Each student in the group must suggest at least on factor that can contribute to poor air quality.

b) Next, each member of the group must give at least one reason why their factor is the most important.

c) Next, as a group, the students must agree on the factor that is most important.

Task

6

optimizing assessment for all - brookings institution...optimizing assessment for all may 2020...

Documents