evaluation policy and guidelines for evaluations · 5 evaluation policy and guidelines for...

IOB

Evaluation policy and guidelines for evaluations

IOB | October 2009 | Policy and Operations Evaluation Department | IOB | October 2009 | Policy and Operations Evaluation Department | IOB | October 2009 | Policy and Operations Evaluation Department | IOB | October 2009 | P

Evaluation policy and guidelines for evaluations

tyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüssstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTeneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTtBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsiagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewY

auloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWtáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKiegMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouP

allinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKmiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoProntoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoP

aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonoanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskou

aramariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PdisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTrpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTngkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHannshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskoutiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkoazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYhanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKabuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouPockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingP

vivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoP

ancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKrobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPshingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonoGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascussinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskou

ortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsselanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskoubatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTtBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsiagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewY

auloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWtáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKiegMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouP

allinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKmiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoProntoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoP

aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonoanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskou

aramariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PdisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTrpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTngkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHannshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskoutiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkoazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYhanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKabuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouPockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingP

vivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoP

ancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKrobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPshingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonoGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascussinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskou

ortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsselanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskoubatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTtBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsiagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewY

auloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWtáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKiegMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouP

allinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetB

Evaluation policy and guidelinesfor evaluations

3

Evaluation policy and guidelines for evaluations

Contents

1 Introduction 4

2 Short history 6

3 Basic principles of IOB’s evaluation policy 10

4 Definitions and criteria for evaluations 14

4.1 Whatisevaluation? 15

4.2 Typesofevaluation 16

4.3 Evaluationcriteria 17

4.3.1 Efficiency 17

4.3.2 Effectiveness 18

4.3.3 Impact 19

4.3.4 Relevance 19

4.3.5 Sustainability 20

4.3.6 Policyconsistency 21

5 Guiding principles and ethical standards 22

6 The evaluation process 26

6.1 Programming 27

6.2 Preliminarystudy 29

6.3 TermsofReference 30

6.4 Deployingresearchers 32

6.5 Cooperatingwithresearchersinpartnercountries 32

6.6 Methodology 33

6.7 Reporting 37

7 Quality control 40

7.1 Internalpeerreview 41

7.2 Referencegroups 42

7.3 Departmentalmeeting 42

7.4 Advisorypanel 43

7.5 Self-evaluation 43

8 Communication and feedback 44

1Introduction

5

Evaluation policy and guidelines for evaluations

ThisdocumentdescribestheevaluationpolicyandguidelinesforevaluationsofthePolicyandOperationsEvaluationDepartment(IOB)oftheDutchMinistryofForeignAffairs.ItplacestheevaluationpolicyinthecontextoftheOrder on PeriodicEvaluation and Policy Information(knownasRPE2006)1,thegovernment-wideframeworkfortheevaluationofgovernmentpolicyingeneral.ItalsoreflectsrecentdecisionsregardingthefunctionofevaluationattheMinistryofForeignAffairs,andtheroleIOBplayswithinit.

IOB’sevaluationpolicyandtheguidelinesmustbeseeninthecontextofachangedenvironment.AnincreasinglycriticaloutsideworlddemandsaclearerinsightintotheresultsoftheNetherlands’foreignpolicy.Thereisgrowingpressureforministerstoaccountnotonlyfortheuseofpublicfunds,butevenmoresofortheresultsachieved.Thisalsoappliestothevariousareasofforeignpolicy.Ifevaluationsaretocontributetothis,theymustbeindependent,public,transparentandaccessible.

Inpublishingtheseguidelines,IOBwishestoprovidetransparencytothirdparties:colleaguesattheMinistryofForeignAffairs,theDutchparliament,organisationsandinstitutionsaffectedbyIOBevaluations,usersoftheevaluationresults,andinterestedcitizens.Ineachevaluationreport,IOBaccountsforthespecificevaluation;thisdocumentclearlypresentsthegeneralprinciplesandguidelinesIOBappliesinternallyinconductingevaluations.

TheguidelinesoutlinedinthisdocumentapplyinprincipletoallevaluationsconductedbyIOB,unlessexplicitlystatedotherwise.ItshouldbeborneinmindthattheNetherlandshasalongtraditionofevaluationinthefieldofdevelopmentcooperation,whichisexpressedthroughinternationalagreements,fora,cooperation,etc.andwhichimpactsonIOB’swork.Thatisnotthecaseforotherareasofforeignpolicy.Becauseitsmandatealsoextendstotheseotherareasofpolicy,IOBisinanexceptionalposition,especiallyinternationally.

TheevaluationpolicyandguidelinesreflecttheopinionsandperspectivesthatunderpinIOB’swork.Inthatsense,thisdocumentdescribestheidealsituationforIOB.Realitywillsometimesdeviatefromthisidealinsomeways.Therewillalwaysbeacertaintensionbetweentheoryandpractice.However,IOBaimstoworkasmuchaspossibleaccordingtotheseguidelines.Itisimportanttorememberthatneitherpolicynorguidelinesarecarvedinstone.Thismustbeseenasagrowthdocument,inthemostliteralsense.Newinsightsmayleadtoitbeingadaptedinthefuture.IOBwillthusperiodicallyreviewitspolicyandtheguidelinesand,ifnecessary,revisethem.Thelatestupdatedversionofthisdocumentwillbeavailableathttp://www.minbuza.nl.

1 Order on Periodic Evaluation and Policy Information 2006, Government Gazette, 26 April 2006, no. 83.

2Short history

7

Evaluation policy and guidelines for evaluations

Inthefirsthalfofthe1970sthebudgetfordevelopmentcooperationincreasedrapidly.Sorapidlythatconcernswerevoicedfromvariousquarters,andespeciallyinthemedia,thatthiswashavinganadverseeffectonthequalityofaid.Asaconsequenceofthisnegativepublicity,theMinisterforDevelopmentCooperationfelttheneedforanindependentopinionondevelopmentpolicy,autonomousofthepolicydepartmentsinvolved.On1July1977theOperationsReviewUnit(IOV)wassetupbyministerialorder.ItwasmadeclearfromthebeginningthatIOVwouldfocusonbilateralcooperation,bothgovernmentalandnon-governmental(insofarasthelatterwasgovernment-funded),butwouldinprinciplerestrictitsactivitiestotheNetherlands’contributiontojointinterventions.Onthebasisofthisrestriction,theministertookthepositionthatIOVcouldconductinspectionswithoutthepermissionoftherecipientcountryororganisation.IOV’sworkprogrammewasdrawnuptogetherwiththepolicydepartmentsandsubmittedtotheministerforapproval.

OutsidetheworkprogrammetheministerandtheministrymanagementwerealsoauthorisedtocommissionIOVtoconductinspectionsonanadhocbasis.However,thisoccurredonlysporadically.FromthestartIOVfocusedonindividualprojects,submittingconfidentialreportstotheminister.Thereportswereconfidentialbecausetheyreferredtoidentifiableeventsandindividuals.

From1979theHouseofRepresentativesoftheDutchparliamentcalledeachyearforIOV’sreportstobemadeaspublicaspossible.Eventually,itwasagreedthatgeneralreportswouldbesenttoparliament,togetherwithsummariesofprojectinspections.ThefirstsummarisedreportthatIOVsenttotheHousewasthe1984GeneralEvaluationofDutchBilateralDevelopmentCooperation.

Inthe1980s,IOV’sareaofworkshiftedfromtheinspectionofindividualprojectstoevaluationsofsectors,policythemesandcountryprogrammes.DuringverbalconsultationswiththeHouseofRepresentativesin1987theministeragreedthatallIOVreportswouldbefullypublicandwouldbesenttoparliament,togetherwithapolicyletterfromtheministercontaininghisorherresponsetothereport.ThereportswereusedactivelyindebatesbymembersofboththeHouseofRepresentativesandtheSenate.Universitiesandotherinstitutionsinthepublicandprivatesectorsalsoshowedalivelyinterestinthereportsoncetheybecameaccessible.

In1996afar-reachingreassessmentofDutchforeignpolicytookplace.Itwaswidelyacknowledgedthatdevelopmentcooperation,foreignpolicyandinternationaltradecouldnolongerbeseenasindependentpolicyareas.Foreignpolicywas‘decompart-mentalised’andthevariousareasofinternationalactivitywereintegrated.Forthepurposeofthereview,IOVpublishedanewsynthesisstudy,summarisingtheresultsoftheevaluationreportsoftheprecedingdecade.Duringthereorganisationthataccompaniedthepolicyreassessment,itwaslogicaltomodifyIOV’smandate.Whereas,beforethereview,IOVwasonlyresponsibleforevaluationsinthefieldofdevelopmentcooperation,itsareaofoperationswasnowextendedtoincludethe

8

2 Short history

wholeofforeignpolicy.Thisimpliedthat,inthefuture,theUnitcouldaddressitselftoallsubjectsthatfellundertheHomogenousBudgetforInternationalCooperation(HGIS).ThechangeinIOV’smandateledtoarevisionofitspositionintheorganisationalstructureoftheministry.ItwouldnolongerbeaccountabletotheDirector-GeneralforInternationalCooperation(DGIS)butdirectlytotheSecretary-General.TheUnitwasalsogivenanewname:inthefutureitwouldbeknownasthePolicyandOperationsEvaluationDepartment(IOB).

InrecentyearsIOB’spositionandrolehasonceagainbeenthesubjectofpoliticaldebateanddiscussionwithintheministry.Manyoftheissuesdiscussedinthe1980swereonceagainontheagenda,includingtheindependenceofevaluation,thepublicnatureofthereports,therelationshipbetweenexternalaccountabilityandinternallearning.Thisledin2008totheMinisterforDevelopmentCooperationreconfirmingtoparliamenttheindependenceofIOBandthepublicnatureofitsreports.However,IOB’sprogrammingauthoritywasrestrictedtodevelopmentcooperation.2Intheotherfieldsofforeignpolicy,primaryresponsibilitynowlieswiththeDirector-Generalconcerned,whocanrequestIOBtoconductevaluations.

2 Letter from the Ministers of Foreign Affairs and for Development Cooperation of 16 October 2007 on implementation of the Boekestijn/Gill’ard motion regarding the Policy and Operations Evaluation Department (IOB) (31200V, no. 9). Report from the Permanent Committee on Foreign Affairs on IOB of 29 May 2008, approved on 25 June 2008.

9

Evaluation policy and guidelines for evaluations

3Basic principles of IOB’s evaluation policy

11

Evaluation policy and guidelines for evaluations

ThePolicyandOperationsEvaluationDepartmentistheevaluationdepartmentoftheMinistryofForeignAffairs.Institutionally,IOBisdirectlyaccountabletotheSecretary-General.Itformspartoftheministry’sAuditCommittee.TogetherwiththeAuditDepartmentandtheInspectionandEvaluationUnit(ISB),IOBformsthecluster‘InControl’.

Internationally,IOBisamemberoftheOECD/DACnetworkofevaluationservices.ItalsotakespartinconsultationsbetweentheevaluationservicesoftheEuropeanUnion,andofthecountriesunitedintheNordic+group.IOBcontributestotheactivitiesofanumberofotherformsofinternationalcooperation,includingtheconsultationcommitteeofGerman-speakingevaluationservices(DACH),theNetworkofNetworksforImpactEvaluation(NONIE)andtheInternationalInitiativeonImpactEvaluation(3IE).

IOB’smandateisasfollows:

The Policy and Operations Evaluation Department meets the need for evaluation of policy and its implementation with regard to all policy areas that are financed by the homogenous budget for international cooperation (HGIS). IOB evaluates the implementation of the Netherlands’ foreign policy, giving particular attention to its relevance, efficiency and effectiveness, the sustainability of the results achieved and the consistency and coherence of the policy.

Inthelightofthismandate,IOBhasthefollowingtasks:

1)toconducthighqualityevaluationsforlearningandaccountabilitypurposes;2)toensurebalancedevaluationprogramminginthefieldofdevelopmentcooperation

andtoadviseonprogramminginotherareasofforeignpolicy;3)tocontributetoimprovementofthequalityofevaluationsconductedby,oronbehalf

of,policydepartments,institutionssubsidisedbytheministry,andmultilateralinstitutionsbydrawingupgenerallyapplicablequalityrequirementsandguidelinesforconductingevaluations,advisingonthedesignandimplementationofindividualevaluations,andconductingsystematicassessmentsoftheseevaluationsorpeerreviewsofotherevaluationservices;

4)tostrengthencapacitytoconductevaluationsinpartnercountriesbysupportingeducationandtrainingandparticipatinginjointevaluations;

5)topromotethedistributionandcommunicationofevaluationresults.

Re1:IOB’smaintaskistoconducthigh-qualityevaluationswithintheframeworkoftheMinistryofForeignAffairs’evaluationprogramme.TheseevaluationscanrelatetoallareasofforeignpolicythatfallundertheHGIS.Inthiscontext,evaluationcomprisescriticalandthoroughinvestigationofapolicyorprogrammeandexplainingtheresults.IOBevaluationsdonotstandinisolationbutareinprinciplepartofthepolicycycle,inwhichplanning,monitoringandevaluationareseenasanintegratedwhole.

12

3 Basic principles of IOB’s evaluation policy

Evaluationisthereforeanessentialcomponentofthe‘knowledgefunction’withintheministry.Knowledgeontheimplementationandimpactofpolicysupportstwoconcretefunctions:

• itprovidesthebasisforaccountingtoparliamentandsocietyatlargeforpolicyandexpenditure(accountabilityfunction);

• itgeneratesinformationforthepurposesofimprovingpolicyformulationandimplementation(learningfunction).

Re2:AsstatedintheRPE2006systematicevaluationmustbeafixedcomponentofeverypolicyprocess.Eachpolicyareamustbeperiodicallyevaluatedatthelevelofoperationalobjectives.TheRPE2006thusformsthebasisofevaluationprogramming.Theprogrammingisalsodeterminedbyspecificrequestsfromparliament,ministersandstatesecretaries,andbudgetholders.WithintheMinistryofForeignAffairs,IOBhasfinalresponsibilityforprogrammingevaluationsinthefieldofdevelopmentcooperationandadvisesonprogramminginotherareasofforeignpolicy.

Re3:IOBisnottheonlyunitattheMinistryofForeignAffairsconcernedwithevaluation.Policyadvisorsalsohavetodealwithevaluations,asthecontactforevaluationprogrammingwithintheirpolicydepartment,asa‘client’commissioninganevaluation,sometimesas(joint)implementer,orasauseroftheresults.Inadditiontoconductingevaluations,IOBthereforehasanadvisoryfunction.ThisentailsformulatingqualitycriteriaandguidelinesforimplementingevaluationsandadvisingontheTermsofReferenceforevaluationsthatIOBdoesnotconductitself.Duringtheannualsettlementoftheprogramme,agreementsaremaderegardingtheextenttowhichIOBwillplayanadvisoryroleinevaluationsconductedbyoronbehalfofministrydepartments.IOBalsoadvisesonthedesignoftheministry’sevaluationfunction,theevaluationsystemandinstruments,andcoordinationwithplanningandmonitoring.InadditionIOBperformsqualitytestsonprogrammeevaluationsconductedbycofinancednon-governmentalorganisations(MFS),withtheaimofstrengtheningtheirevaluationcapacity.

ForthemultilateralchannelIOBisinvolvedininternationalreviewsoftheevaluationservicesofrelevantinstitutions.

Re4:Strengtheningevaluationcapacityinpartnercountriesisanimportantthemeindevelop-mentcooperation.Independentevaluationcapacityinpartnercountriesisindispensableiftheaimofestablishingasystembasedonmutualaccountabilityistobeachievedinreality.IOBseesitasitstasktocontributetothisbysupportinginternationalinitiativesaimedatstrengtheningevaluationcapacity.Inaddition,wherepossibleandsafeguardingitsowncriteriaforindependenceandquality,IOBworkstogetherwithresearchersfrompartnercountriesinconductingevaluations.

13

Evaluation policy and guidelines for evaluations

Ad5:Evaluationsgenerateknowledgeonthebasisofexperiencefromthepastregardingwhatworksandwhatdoesnot,andespeciallywhy.IOBconsidersitimportantthattheresultsofevaluationsaredistributedaswidelyaspossible.Toachievethis,itdiscussesthefindingsinstaffmeetings,withministersandstatesecretaries,andwithothernationalandinternationalpartiesinvolvedintheevaluations.Inaddition,wherepossibleandwhereitdoesnotconflictwithitsownresponsibilities,IOBwilloffersupportinpreparingpolicyresponsesanddescriptionsofevaluationfindingsintheMinistryofForeignAffairs’AnnualReports.

4Definitions and criteria for evaluations

15

Evaluation policy and guidelines for evaluations

4.1 What is evaluation?

Thereisawidevarietyofdefinitionsoftheconceptofevaluation.Inthe1980sand1990s,theOECD/DACworkedonasharedframeworkofconceptsforevaluationpracticesinthefieldofdevelopmentcooperation.TheresultingdefinitionofevaluationservesasabasisforIOB’swork.

‘Anevaluationisanassessment,assystematicandobjectiveaspossible,ofanongoingorcompletedproject,programmeorpolicy,itsdesign,implementationandresults.Theaimistodeterminetherelevanceandfulfilmentofobjectives,developmentalefficiency,effectiveness,impactandsustainability.Anevaluationshouldprovideinformationthatiscredibleanduseful,enablingtheincorporationoflessonslearnedintothedecision-makingprocessofbothrecipientsanddonors.’3

TheOECD/DACdefinitionreflectsanunderstandingofevaluationthatisbroadlysupportedintheinternationalcommunityofevaluators.Thedefinitionisintendedforevaluationsinthefieldofdevelopmentcooperation.SinceIOB’sworkextendsfurtherthandevelopmentcooperation,conceptslike‘developmental’and‘recipientsanddonors’willnotapplytoallIOBevaluations.

Evaluationisdifferenttoreviewsandmonitoring.TheOECD/DACGlossarydescribesareviewasanassessmentoftheperformanceofanintervention,periodicallyoronanadhocbasis.ForIOBthemaindifferencebetweenareviewandanevaluationisthatareviewisconsideredamanagementinstrumentforoperationalmonitoringoftheimplementationofanactivity,andthereforefocusesonoperationalaspectsandrecordingresults,whileevaluationisamorein-depthassessmentoftheaims,implementationandresultsofanintervention,withthequestionofattributionasitsmainfocus.

Monitoringisaninstrumentofinternalmanagement.Itaims,onthebasisofafixedsetofindicators,toprovidearegularinsightintotheprogressmadeinimplementingactivities.Monitoringallowsabudgetholderortheimplementerofaninterventiontostayuptodateoninputssuppliedandactivitiesperformed(outputs).Monitoringshowswhetherintendedactivitieshaveactuallybeencarriedout.Good,systematicmonitoringisofgreatimportanceforevaluations,asitprovidesasignificantpartofthedataonwhichevaluationisbased.

3 DAC principles for Aid Evaluation, OECD/DAC, 1991 (the definition was also included in the DAC Glossary

of Key Terms in Evaluation and Results-based Management in 2002).

16

4 Definitions and criteria for evaluations

4.2 Types of evaluation

TheOrderonPeriodicEvaluationandPolicyInformation2006(RPE2006)prescribesasystemofevaluationbasedontheso-calledoperationalobjectives,OD.Thesystementailsperiodicalevaluationoftheseoperationalobjectives.

TheRPE2006distinguishesbetweenpolicyreviewsandex-postimpactstudies.Bothconceptsdeviatefromtheevaluationterminologyusedinternationally,butaretheequivalentofthemorecommonterms‘policyevaluation’and‘impactevaluation’.Thestartingpointisthatpolicyreviews,whicharelinkedtotheoperationalobjectives,areasfaraspossiblefedbyimpactstudies.Inprincipletheyresemblesynthesisstudies.IOBgivesprioritytothesetypesofevaluation,where–especiallyinthefieldofdevelopmentcooperation–policyreviewsareprecededbyaseriesofex-postimpactstudies(impactevaluations).

OnthebasisofthesystemprescribedintheRPE2006,IOBimpactevaluationswillgenerallyfocusoninterventions(rangingfromprogrammestovariousformsofbudgetsupport).Policyevaluations(orreviews)addressthemesorsectors.Inaddition,integratedforeignpolicycanalsogiveriseto‘decompartmentalised’evaluationsofcountryorregionalpolicies.

Besidesthese–usuallyextensive–evaluations,IOBcanalsoconductshortinternalevaluationsthatarelimitedinnatureandnecessarytomonitortheprogressofprogrammes.

PartoftheevaluationsinIOB’sprogrammingfocusessolelyonDutchprogrammesandactivities.ItislogicalfortheseevaluationstobeconductedbyIOB.However,itisbecomingincreasinglycommontoworktogetherwithinternationalpartnersinthefieldofdevelopmentcooperation.Projectsbyindividualdonorsaregraduallybeingreplacedbyjointfinancingofgovernmentpolicyinpartnercountries.Itisthenalsologicalthatdonorsgivegreaterpreferencetothejointprogrammingandimplementationofevaluations.Incaseswheredonorsworktogether,forexampleinprovidingjointsupportforsectorpolicyorotherfar-reachingformsofjointfinancing(suchasbasket-funding),IOBevaluationswillinprinciplebeconductedincooperationwithotherdonors,aslongasIOB’scriteriaforindependenceandqualityaremet.Thenatureofthecooperationcanvaryfromactualparticipationbyseveraldonorsinanevaluation,agreeingonadivisionoftaskswithinasharedTermsofReference,toagreementonthedivisionoflabourbetweendifferentthemes,sectorsorcountries.

17

Evaluation policy and guidelines for evaluations

4.3 Evaluation criteria

Evaluationcallsforatailor-madeapproach.Thatdoesnotmean,however,ignoringqualitycriteriaorabandoningconsistencyinapproach.Inallevaluations,theevaluationcriteriamustbeappliedinanunambiguousand,aboveall,transparentway.Here,inparticular,standardisationisparamounttoenableevaluationsandtheirresultstobecompared.TheevaluationcriteriaappliedbyIOBarelargelyidenticaltothosedrawnupbytheOECD/DAC.TheOECD/DACdefinitionofevaluationspecifiesfivecriteriaforevaluationofdevelopmentinterventions:efficiency,effectiveness,impact,relevanceandsustainability.Theevaluationcriteriarefertodifferentresultlevelsfoundedonthelogicalframeworkorresultchainsbasedonit.Together,thesecriteriaprovideaframeworkwithinwhichresearcherscanpresenttheirfindingsinanorderedandcoherentway.Theextenttowhichequalattentioncanorwillbepaidtoallcriteriawillvaryperevaluation.Furthermore,additionalcriteriacanbeapplied:evaluationof‘decompartmentalised’policy,forexample,mayrequirethatIOBdevotesexplicitattentiontopolicyconsistency.

4.3.1 Efficiency Efficiencymeasureshoweconomicallyresources(inputs)andthewaytheyareappliedareconvertedtodirectresults.Theconceptthereforeillustratestherelationshipbetweeninputandoutputandreferstoalevelofresultsthattheimplementercancheck.

Inevaluatingefficiency,boththequantityandqualityofinputsandoutputsareassessed.Themosteconomicalorcost-effectiveinputisnotalwaysthemostappropriateandtherecanbesignificanttradeoffsbetweenthequantityandqualityofoutputs.Assessingefficiencyalsocallsforcomparisonwithalternativeapproaches,withwhichthesameoutputscanbeachieved.Thisisbynomeansalwayseasybecausethereareoftennopreviouslyformulatedbenchmarksandtheymighthavetobedevelopedonthebasisofpracticalexperience.Thatismorelikelyinthecaseof,forexample,interventionsinvolvingconstructionactivitiesthanofthoseaimingtobringaboutsocialorinstitutionalchange.

Thefollowingelementsshouldbetakenintoaccountwhenevaluatingefficiency:

• Haveappropriateinputsbeendeployedatthelowestpossiblecost?• Haveactivitiesbeenconductedinanuncomplicatedmanner?• Havedecisionsbeenmadeattherightlevelandhasbureaucracybeenavoided

asfaraspossible?• Haveoverheadcostsbeenkepttoaminimum?• Hasduplicationbeenavoided?• Haveconflictsduringimplementationbeenpreventedorsolved?• Haveoutputsbeenachievedwithintheplannedperiodandbudget?

18

4 Definitions and criteria for evaluations

4.3.2 Effectiveness Effectivenessrelatestotheextenttowhichthedirectresultsofinterventions(output)contributetothesustainableachievementofpolicyobjectives(outcome).Policyobjectivesaretheobjectivesthatinterventionsareintendedtoachieveandshould,inprinciple,beformulatedintheprojectdocument.Insomecasespolicyobjectivescanbeidenticaltotheoperationalobjectivesspecifiedinthebudgetorspecificcomponentsofthem.

Aninterventionisconsideredeffectiveifitsoutputshavemadeademonstrablecontributiontoachievementoftheintervention’sintendedobjectives.Efficiencyandeffectivenessrefertotwosuccessivelevelsintheresultschain.Apositiveassessmentatoneleveldoesnotautomaticallyimplyapositiveassessmentattheother.Aninterventioncanbeefficientwithouthavinganeffect,andaneffectiveinterventioncanbeinefficientlyimplemented.

Theeffectsofaninterventionarebeyondthedirectcontroloftheimplementer.Interventionsgenerateoutputswhichareassumedtogeneratethedesiredeffects.Evaluatingeffectivenessaimstoestablishthiscausality.Thecoreproblemisthatitwouldbeidealtoknowwhatwouldhavehappenediftheinterventionhadnotoccurred(thecounterfactual).Onlythencanobservedchangesbeattributedtotheinterventionandisitpossibletospeakoftheneteffects.Inotherwords,thereisaproblemofattribution.Aslongasotherfactorshaveavisibleimpactontheobservedchanges,theproblemisrelativelysimpletosolve.Itismoredifficultwheninfluentialfactorsarenotimmediatelyvisible(the‘unobservables’).Researcherscannotknowwhattheydonotknow.Evaluationstrytofindasolutiontothisproblem(see6.6).

Inpractice–especiallyinthecaseoflargeandcomplexinterventions,butalsowithinterventionsthataimtobringabouteffectsthataredifficultorimpossibletoquantify–evaluationsoftenmakeuseofplausibilityargumentation.Itisthenimportantthattheargumentistransparentandcredible.

Inessence,threestepscanbedistinguishedintheevaluationofeffectiveness:

• Themeasurementofchangesintheeffectvariablesincomparisontothesituationatthestart(baseline);

• Attributionoftheobservedchangestotheintervention;• Assessmentofchangesobservedandattributedtotheinterventionintermsofthe

objectives.

19

Evaluation policy and guidelines for evaluations

4.3.3 Impact Thebroadconsensusinthecommunityofevaluatorsabouttheconceptsofefficiencyandeffectivenessislessevidentwhenitcomestoimpactandrelevance.

TheOECD/DACdefinesimpactasallsignificanteffectsproducedbyadevelopmentintervention,directlyorindirectly,intendedorunintended,ontheultimatestakeholdersandthirdparties.Whileeffectivenessrefersonlytotheintendedeffectsofanactivity,inthisterminologyimpactreferstoallpossibleeconomic,social,political,technicalandecologicaleffectsatlocal,regionalornationallevelthathaveadirectorindirecteffectonthetargetgrouporotherparties.

Inthestrictercontextofthelogicalframework,theconceptofimpactusuallyreferstothehighestlevelofresults.Thequestionisthentowhatextent,withtheaidofoutputs,theeffectsachieved(outcomes)havecontributedtobroader,morefar-reachingobjectives.Thesecanbeoperationalobjectivesorobjectivesatahigherlevel,suchasmoregeneralpovertyreduction.

Intheinternationalliteratureonimpactevaluations,theconceptofimpactisusedfortheneteffectsattributedtotheintervention,i.e.atthelevelofeffectiveness.

Oneinterpretationofimpactisnotbydefinitionbetterthananother.Itisimportantthatconceptsaremadeexplicitinevaluationreports.Intheinterestofmutualcomparisonitisdesirableinex-postimpactstudiestousetheinterpretationofimpactevaluationsmostcommonlyappliedintheinternationalliterature.InthisrespectIOBfollowstheinstructionsintheRPE2006.

4.3.4 Relevance Theconceptofrelevanceisalsosubjecttodifferentinterpretations.TheOECD/DACdefinesrelevanceastheextenttowhichtheobjectivesofadevelopmentinterventionareconsistentwithbeneficiaries’requirements,countryneeds,globalprioritiesandpartners’anddonors’policies.Thatmakesrelevanceacontainerconcept,withinwhichnochoicesaremade.Thisdefinitionisunsatisfactoryforseveralreasons.First,theassessmentofwhetheraninterventionisalogicalconsequenceoforiscompatiblewithpolicyisinessenceanexantequestionthatshouldbepartoftheapprovalprocedure.Second,donorpolicy,thepartner’spolicyandtheneedsofthetargetgroupdonotnecessarilyallperfectlycoincide.Thatwouldimplythataninterventioncanberelevantandirrelevantatthesametime.Third,policyprioritiescanchangeovertime,suchthataspecificinterventionisrelevantatonemomentandnotthenext,orviceversa.Lastly,thisdefinitionimpliesthattherelevanceofthepolicyitselfisnotupfordiscussion.Inotherwords,itexcludesproblemswithpolicyitself.

Inthelogicalframework,relevancereferstotheextenttowhichtheeffectsofinterventionsmakeasustainablecontributiontoachievingtheultimateobjective(theimpact).Aninterventionhasbeenvaluable,orrelevant,totheextentthatithas

20

4 Definitions and criteria for evaluations

generatedeffectsthatbringachievementoftheultimatedevelopmentobjectivecloser.IOBwillapplytheconceptofrelevanceinitsevaluationsinthisrespect.

4.3.5 Sustainability Sinceanactivitycanhardlybeconsideredeffectiveiftheeffectithasachievedisnotlasting,sustainabilityisinfactanaspectofeffectiveness.4Atthesametimetheconceptofsustainabilitycomprisesavarietyofdimensionsthatarerelevantforassessinganintervention.Inthedonorcommunitythesustainabilityofinterventionsislinkedtosevenfactors,whichcanbeusedtoanalyseinterventions.Whichfactorsorcombinationsoffactorsareusefulforassessingsustainabilitywilldependonthenatureoftheintervention.

• Theinvolvementoftherecipientisusuallyrecognisedasafactorthataffectsthesuccessofanintervention.Involvementisexpressedthroughagreementonobjectivesandtheprovisionofhumanandmaterialresources;

• Whendeployingtechnologyitisimportanttotakeaccountofthefinancialandinstitutionalcapacitiesoftherecipientandthedegreetowhichtheconditionsformaintenanceandrenewalareguaranteed;

• Interventionscanhaveaneffectontheenvironmentintheshortorlongterm.Whileshort-termeffectsareusuallyenvisagedandarepreventedormitigated,itisespeciallyimportanttoincorporatelong-termenvironmentaleffectsintheassessment,astheycanneutralisepositivechangesorhaveotherharmfulconsequences;

• Thedegreetowhichchangesaresustainabledependsverystronglyonsocialandculturalfactors,whichareoftennotdirectlyvisible.Studyingperceptionscanprovidebetterinsightintothisdimensionofsustainability;

• Institutionalandcapacitydevelopmentplayanimportantroleindeterminingsustainability.Whenassessinginterventionstheattentiondevotedtotheseaspectswillhelpdeterminetheextenttowhichsustainabilityisachieved;

• Financialandeconomicsustainabilityreferstothedegreetowhichmeasureshavebeentakentoguaranteethatactivitiescancontinueandcompletedworkscanbemaintainedinthefuture;

• Interventionsalwaystakeplaceinawidernatural,political,economicandsocio-culturalenvironment,whichcannotbeinfluencedinanywayatallbutwhichitselfhasastronginfluenceoneffectivenessandsustainability.Changesinpoliticalstability,economiccrises,ornaturaldisasterscanhaveadecisiveeffectonthesustainabilityofinterventions.

4 In the context of humanitarian aid this is, of course, different. It usually entails short-term,

punctual interventions, the effects of which it is hoped will only be necessary for a brief period.

21

Evaluation policy and guidelines for evaluations

4.3.6 Policy consistency Thisconceptreferstotheabsenceofinherentcontradictionsbetweendifferentpolicyareas.ConsistencyisparticularlyimportantforIOBintheevaluationofdecompartmentalisedpolicy.Developmentobjectivescanconflictwithinternationaltradeinterestsorforeignpolicyconsiderations.Thatiswhyitisimportant,particularlyincaseswhereobjectiveshavetobeachievedinthecontextofdecompartmentalisedpolicy,todeterminetheextenttowhichinterventionsperformedfromdifferentperspectivesarecoherentwithanddonotobstructeachother.Theaimistohelpachieveintendedobjectiveswithouttheeffectsofoneinterventionneutralisingthoseofanother.

5Guiding principles and ethical standards

23

Evaluation policy and guidelines for evaluations

IOB’sguidingprinciplesreflectthecorevaluesonwhichitsevaluationsarefounded.Theyareinterrelatedandmutuallyreinforcingand,assuch,theymustbeseenasprovidinganoverallframework,withwhichallaspectsofIOB’sevaluationworkmustbeconsistent.Theprinciplesdeterminewhatcanbeexpectedfromaninspectorandotherresearchersinvolvedinanevaluation,fromevaluationprocessesandproducts,andfromthemethodology.TheyareconsistentwiththeOECD/DAC’sPrinciplesforEvaluationofDevelopmentAssistance(1991)andEvaluationStandards(draft2009).

AllIOBevaluationsmustbeconsistentwiththefollowingguidingprinciples:

Independence Thejudgementsoftheresearchermustnotbeinfluencedbyexternalpressureoraconflictofinterests.Organisationsorinstitutionsthatarecommissionedtoperformanevaluationandmembersofevaluationteamsmaynothavebeenpreviouslyinvolvedintheinterventionsthataretobeevaluatedorinformulationofthepolicyonwhichtheyarebased.

Objectivity Theevaluationmustbebasedonverifiablefacts.Theresearchermustmakeeveryefforttoensurethatthedataonwhichtheevaluationisbaseddoesnotcontainanyinaccura-cies.Thepresentationoffactsshouldbeclearlyandrecognisablydistinguishedfromjudgments.

Transparency Theevaluationmustbeconductedinawaythatcanbefollowedclearlybystakeholdersandthirdparties.Thequestionstobeaddressed,thedatabase,theresearchmethods,findingsandconclusionsmustbepresentedinthereportinaclearandaccessibleway.

Validity TheevaluationmustmeasurewhattheTermsofReferencespecifiesasrequiringmeasurement,andmustdothisinawaythatthereadercanunderstand.

Reliability Theresultsoftheevaluationshouldbestable,i.e.thatarepeatoftheevaluationwillproducethesameresultsandconclusions.

PartnershipInthecaseofevaluationsinthefieldofdevelopmentcooperation,wherepossibleandinsofarasitdoesnotconflictwithotherprinciples,partnersfromtherecipientcountrieswillbeinvolvedintheimplementationofevaluations.Everyeffortwillbemadetocooperatewithindependentresearchersfromthepartnercountries.

24

5 Guiding principles and ethical standards

Feasibility Evaluationsareonlyconductedifthedatabaseisaccessibleandtheassociatedmethodologyisavailable.

Correctness Theevaluationmustnotcauseharmtoindividuals.

Cost effectiveness Theevaluationmustbeconductedatthelowestpossibleprice,takingaccountoftheotherguidingprinciples.

Usability Theevaluationmustresultinconclusionsandpointsforattentionthatarevaluablefortheuseroftheevaluationbysuggestingactionsthatliewithintheuser’scapacity.

Evaluationisaformofresearchthatentailsajudgment.Itdiffersfrompureacademicresearchinthatitalwaysresultsinanopinionontheresultsofconcretehumanactions.Researchersthereforehaveagreatresponsibilitytowardsstakeholdersinvolvedintheevaluation.ThisappliesequallytoIOBinspectorsandtoexternalresearcherscontractedbythem.Researcherscanthereforebeexpectedtoadheretointernationallyacceptedethicalstandardsforevaluations.

ThefollowingethicalstandardsapplytoIOB’swork5:

Respect and understanding for other cultures Manyevaluationsareconductedinadifferentculturalenvironmenttothatoftheresearchers.Theycanthereforebeexpectedtocomplywithlocalculturalcodesduringinterviewsandotherformsofdatacollection.

Universal values Someculturalcustomsandpracticesmaybeincompatiblewithfundamentalrightsandfreedoms.Althoughresearchersshouldinprincipleshowrespectforothercultures,theymustnotignoretheeffectofcertainculturalvaluesongenderrelationsorminoritiesandotherspecificgroups.InsuchcasestheyshouldactinaccordancewiththeUniversalDeclarationofHumanRightsandtheassociatedinternationalagreements.

5 The standards are largely based on the evaluation policies of the Danish International

Development Agency (DANIDA) and the Canadian International Development Agency (CIDA).

25

Evaluation policy and guidelines for evaluations

Gender Evaluationsshouldbecarriedouttakingaccountofthedifferentrolesperformedbymenandwomenandtheaimofachievinggenderequality.Inpolicyareaswheregenderdifferencesareadimensionoreffectofthepolicypursued,theyshouldbetakenintoaccountinthedesignandimplementationoftheevaluation.

Anonymity and confidentiality Researchersareexpectedtorespectrequestsfrominformerstotreattheinformationtheyprovidewithconfidentialityandtotakestepstoensurethatconfidentialinforma-tioncannotbetracedbacktothesource.

Responsibility Inmanycountriesacriticalstanceregardingthegovernmentcanhavesevereconse-quencesforresearchersfromthecountryitself.Inevaluationsinvolvinglocalresear-chersitisimportantthatsuchrisksberecognisedintimeandstepstakentoavoidthem,forexample,bymakingclearagreementsaboutwhoisresponsibleforthecontentofthereport.

Omissions Researchersmayobtaininformationthatisnotdirectlyimportantinansweringthequestionsaddressedintheevaluation,butisneverthelessrelevantinthelightoftheaccountabilityandlearningfunctionsoftheevaluation.Insuchcases,theclient(inthelastinstance,theIOBDirector)willbeconsultedtodeterminewhetherthisinformationshallbeincludedinthereportandinwhatform.

Assessment of individuals Evaluationsarenotassessmentsofindividuals.Yetevaluationreportscanincludefindingsonleadershipandthemanagementorotherqualitiesofimplementersthatcomeclosetopersonalassessments.Researchersshouldthenmakeeveryefforttopresentthefactsofeasilytraceablecasesinanabstractform.Ifindoubt,theyshouldconsultwiththeIOBDirector.

Evidence of malpractice Evaluationsarenotinvestigationsintolegitimacy.Neverthelessitispossiblethatmalpracticescometolightduringevaluations.InthatcasetheresearchershouldsubmitthecasetotheIOBDirector.Ifthereissufficientreasontodoso,theAuditDepartment(ACD)willbeinformed.

6The evaluation process

27

Evaluation policy and guidelines for evaluations

6.1 Programming

Thequalityofevaluationbenefitsfromacarefulandtransparentdecision-makingprocess.Thisprocessstartswiththeprogrammingofevaluations.IOBhasfinalresponsibilityforprogrammingevaluationsinthefieldofdevelopmentcooperation.ThatmeansthattheIOBDirectorapprovestheprogrammingafterithasbeendiscussedwiththeFinancialandEconomicAffairsDepartment(FEZ)andtheotherpolicydepartmentsinvolved.Withrespecttotheotherareasofforeignpolicy,IOBadvisesontheprogramming,whichisapprovedinthemeetingoftheSecretary-GeneralandtheDirectors-General.IOBappliesthebasicprincipleslaiddownintheRPE2006forsystematisationoftheprogramming.Theevaluationprogrammingdistinguishesbetweenthreetypesofevaluation:policyreviews(policyevaluations),ex-postimpactstudies(impactevaluations)andotherevaluations.Policyreviewsandex-postimpactstudiesfallundertheRPE2006.TheRPEprescribesthateachpolicyareamustbeperiodicallysubjectedtoapolicyrevieworpolicyevaluationatthelevelofoperationalobjectives.Apolicyreviewshouldmakeasmuchuseaspossibleofex-postimpactstudies.PolicyreviewsandimpactstudieslargelydeterminetheIOB’sresearchagenda.

InadditiontotheRPE2006therecanbeotherreasonsforincludingevaluationsintheprogramming.

First,therearepoliticalconsiderations.Atcertaintimespolicyissuescanbethesubjectofpoliticalorsocialdebate.Itisthenimportanttofeedthedebatewithinsightsobtainedthroughthoroughandindependentevaluation.

Policyrelevancecanalsoplayarole.Thisisdeterminedbytheprioritygiventoanissueandthemomentinwhichtheevaluationiscarriedout.Ifitisclearwhenresultsarerequiredfortheformulationofnewprogrammesorpoliciesandanevaluationcanbeconductedwithinthatperiodwithoutlossofquality,IOBensuresthattheevaluationisprogrammedandimplementedintime,irrespectiveofthescaleorcomplexityofthesubject.

Inthefieldofdevelopmentcooperationthereisinternationalpressuretoimprovecoordinationbetweentheevaluationprogrammingofindividualdonorsandtoconductjointevaluationsasfaraspossible.Jointevaluationsincludethosecarriedouttogetherwithevaluationservicesintherecipientcountries,multidonorevaluations,andvariouscombinationsofthetwo.Workingtogetherwithevaluationservicesinrecipientcountrieshelpsstrengthentheirevaluationcapacityandimprovemutualaccountability.ThechoiceofwhichjointevaluationsIOBparticipatesinwilldependoninitiativesdevelopedduringinternationalconsultations,opportunitiestoenterintostrategicallianceswithlike-mindedevaluationservices,theextenttowhichIOB’scriteriaforindependenceandqualityareguaranteed,andwhethertheycanbefittedintoIOB’sexistingprogramming.

28

6 The evaluation process

IOBalsoconsidersitimportanttoallowspaceinitsprogrammingforadhocrequests.ThesemayberequestsfromwithintheministryforIOBtoconductshortevaluationsofalimitednaturetodeterminetheprogressofprogrammesortosupportthedecision-makingprocess.Inaddition,ministers,statesecretariesorparliamentcanrequestIOBtoconductspecificevaluations.IOBmakeseveryefforttocomplywiththeserequests.

Theevaluationprogramminginthefieldofdevelopmentcooperationrelatesnotonlytobilateralcooperationbutalsotothenon-governmentalandmultilateralchannels.Inthecontextofthecofinancingsystem(MFS)IOBconductssystematicassessmentsofprogrammeevaluationscarriedoutundertheresponsibilityoftheco-financednon-governmentalorganisationsthemselves.Inaddition,IOBselectsanumberofthemesorcountriestoconductitsownevaluations.Todate,IOBhasbeensomewhatreservedinprogrammingevaluationsofmultilateralcooperation.Togetherwithotherdonors,IOBhasencouragedmultilateralinstitutionstoimprovethequalityoftheirownevaluationsthroughpeerreviews.Inthefuture,wherepossible,IOBwillparticipateinjointdonorinitiativesaimedatamoresystematicevaluationofmultilateraldevelopmentcooperation.

ThedecompartmentalisationofforeignpolicyandthesubsequentbroadeningofIOB’smandatemeanthatIOBevaluatesnotonlyinterventionsinthefieldofdevelopmentcooperation,butalsoactivitiesinotherareasofforeignpolicy.Itisnotunusualforotherforeignpolicyinstrumentstobedeployedtoaddressaspecifictheme,alongsidedevelopmentcooperation.Thiscanbegoodreasontoconducta‘decompartmentalised’evaluation.Inmanycases,onlytheMinistryofForeignAffairsisinvolvedinsuchanevaluation,butnotalways.Therearealsoissuesofforeignpolicyinwhichotherministriesplayaprominentrole.ForIOBthatisreasontoexplorethepossibilitiesforconductingevaluationsjointlywithotherministries.Insuchcases,andifIOB’sconditionsregardingindependenceandqualityaremet,theseevaluationswillbeincludedintheprogramming.

29

Evaluation policy and guidelines for evaluations

6.2 Preliminary study

Onceasubjectforevaluationhasbeenincludedintheprogrammingapreliminarystudycanbeinitiated.Especiallyinthisinitialphase,anumberofimportantdecisionshavetobemadethatwillhavefar-reachingsignificanceforthefurtherprogressoftheevaluation.Theinspectorwillhavetobecomefamiliarwiththepolicyareatobeevaluatedandtheavailablesourcesofinformation,andformsomeideaofthequestionstobeaddressedintheevaluationandthemostadequateapproach.Withlarge,complexevaluationstheinspectorcandrawupastart-upreport.Thereportdescribesthereasonfortheevaluationandtheobjective.Itcanalsogiveabriefdescriptionofthepolicyareatobeevaluatedandformulatecorequestionsforthepreliminarystudy.Itcansuggestwhichtypeofevaluationwouldbemostappropriateandhowitmightbeorganised(theexpectedduration,whetherfieldmissionsareconsiderednecessary,andwhenandhowexternalconsultantsmightbeneeded).Lastly,thestart-upreportcanofferaninsightintotheexpectedproductsoftheevaluationandthebudget.

Start-upreportsarediscussedwiththeIOBmanagementandtheinternalpeerreviewersdesignatedfortheevaluation(see7.1).Afterdecisionshavebeenmadeontheabovementionedmainpoints,thepreliminarystudywillmoveontodraftingdefinitiveTermsofReferencefortheactualevaluation.

Thepreliminarystudycomprisesanumberofactivities:

Reconstruction of policy theory and intervention logic Theextenttowhichasubjectcanbeevaluatedisimprovedbyclearlyformulatedpolicyprinciplesandprogrammeobjectives,adescriptionoftheactivitiesundertaken,andalogicalrelationshipbetweenpolicy,objectivesandactivities.Practiceshowsthatthesecriteriaarebynomeansalwaysmet.Generallyspeaking,thisappliestobothdevelopmentcooperationandotherareasofforeignpolicy.Especiallyatthelevelofoperationalobjectives,policyisoftenbroad-rangingandnotveryspecificallyformulated,aimsandeffectsarefrequentlyunclearandthelogicofinterventionsisnotsufficientlyexplicit.Insuchcases,onthebasisofthepreliminarystudy,inspectorsthemselveswilldevelopapolicytheoryandareconstructedinterventionlogicbasedonit.Theywillthenpresentthetheoryandthelogictothepolicymakertodeterminewhethertheyareinlinewithwhatthelatterhadinmind.

Defining the evaluation objectPolicythemesandsectorsareoftenbroadlyformulatedandcoverawidevarietyofactivities.Thisappliesinparticularattheleveloftheoperationalobjectivesspecifiedintheministry‘sbudgetdocument.Thepreliminarystudyshouldthereforeestablishthelimitsoftheevaluationobject–alsointhecaseofwide-rangingevaluations–suchthatithasaclearfocusandthusremainsfeasible.Thisoffersabetterguaranteethatthefocusoftheevaluationremainsclearandthatonlythosequestionsareformulatedthat

30

6 The evaluation process

canreasonablybeexpectedtobegivenavalidanswer.Establishingthelimitsofanevaluationrelatestothetimedimension,thegeographicaldimensionandthenatureoftheactivities.Itisbasedonconversationswithrelevantinformants(stakeholders,externalexperts)andonliteratureandfilestudies.

Identifying result indicators Animportantcomponentofreconstructingpolicytheoryisformulatingaresultschain,distinguishingvariouslevelsofresults.Measurableresultindicatorsareidentifiedforeachoftheresultlevels,toenableperformancetobeassessedatthatlevel.

Identifying data sources Identifyingthescaleandqualityofexistingsourcesofinformation(files,literature,informants,statisticaldatabases)andtheopportunitiesforgatheringsupplementaryinformationdeterminetheextenttowhichthesubjectcanbeevaluatedandhowtheevaluationquestionsareformulated.

Formulating the research questions TheresearchquestionsincludedintheTermsofReferencearedeterminedbytheultimateaimoftheevaluation.Theyrelatetotheself-evidentneedforinformationonresultsachievedatthelevelofoutputsandeffects.Thisinformationisofcoursevaluableforboththeaccountabilityandlearningfunctionofanevaluation.Inaddition,supplementaryquestionsrelatingtoaspecificlearningneedcanbeformulated.

Determining the method of evaluation Thechoiceofresearchmethodsdependsonalargenumberoffactors.Itwillbeinfluencedbythenatureofthesubject,thetimeandbudgetavailablefortheevaluation,datathatisalreadyavailableandneedstobecollected,andthenatureoftheresearchquestions.

6.3 Terms of Reference

TheTermsofReferenceestablishtheconditionsunderwhichtheevaluationwillbeconductedandformtheresultofthepreliminarystudy.TheToRmustcontainatleastthefollowingcomponents:

• thereasonandjustificationfortheevaluation:ashortexplanationoftherelevanceoftheproposedevaluation;

• the(external)aimoftheevaluation;• thecorequestions:formulationofthecentralresearchquestionandthesecondary

questionsderivedfromit;• thedemarcationandscopeoftheevaluation:explanationofhowthefieldof

31

Evaluation policy and guidelines for evaluations

researchwillbedemarcatedintermsoftheme,geographicallocationandtime;• approachandchoiceofmethodology,includingfocusonsubsidiarystudies:

descriptionofthedesignoftheevaluation,theindicators,informationsourcesandresearchtechniques;

• representativeness:explanationofhowandtheextenttowhichrepresentativenessplaysaroleintheevaluation;

• organisation:explanationoftheorganisationoftheevaluation;deploymentofexperts,organisationofworkshops,missions,etc.;

• products:indicationoftheexpectedsubsidiaryproducts,includingsubsidiarystudiesoradatabase.Alsoanindicationofwhetherthereportwillbepublishedorwhethertheevaluationisinternal;

• timepath:thetimepathfortheevaluation;• budget:aspecifiedoutlineofthecosts.

IftheTermsofReferenceareforapolicyreviewthequestionsprescribedbytheRPE2006mustalsobetakenintoaccount.Policyreviewsmustaddressthefollowingquestions:

• Whatproblemgaverisetothepolicy?Doestheproblemstillexist?Whatisthecauseoftheproblem?

• Whydoesthegovernmentconsidertacklingtheproblemasitsresponsibility?Whyistheresponsibilityatcentralgovernmentlevel(andnotatlocalorEUlevel)?Whatformdoesthatresponsibilitytakeandwhy?

• Whatobjectivehasthegovernmentformulatedtosolvetheproblem?• Whatinstrumentsarebeingdeployed?Howcoherentarethey?Isthereoverlap?

Whatisknownabouttheimplementationofthepolicyandtheefficiencyofoperationalmanagement?

• Whatistheeffectoftheinstrumentsontheformulatedobjectives(solutiontotheproblem)?Doinstrumentsinotherpolicyareasalsohavesignificanteffectontheformulatedobjectives?Whatarethemainpositiveandnegativeside-effects?

• Howarethebudgetsdetermined?Onwhatbasis?

ThedraftTermsofReferencearenotonlyputontheagendaoftheIOBdepartmentalmeetingbutshouldalsobediscussedinadvancewithdirectstakeholdersandexternalexperts(preferablyintheformofareferencegroup,see7.2),andwiththeinternalpeerreviewers.Thedegreetowhichandthewayinwhichcommentsaredealtwithmustbecleartoallparties.TheTermsofReferenceforanevaluationareapprovedbytheIOBDirector;ifthesubjectoftheevaluationliesoutsidedevelopmentcooperation,theToRwillbeapprovedbytheDirector-Generalconcerned.

TheTermsofReferenceformthebasisonwhichabudgetwillbereservedfortheevaluation.Italsoprovidesaframeworkforcontractingexternalresearchers.ByapprovingtheTermsofReference,theIOBDirectoralsoapprovesthebudgetincludedintheToR.

32

6 The evaluation process

6.4 Deploying researchers

TheIOBinspectorbearsfullresponsibilityforsettingup,conductingandcompletingtheevaluation.Inmostcases,theinspectorwilldeployresearcherstoconducttheevaluation.Themomentatwhichtheywillbedeployedwillvary,aswilltheextenttowhichtheyoperateautonomously.

Researcherscanberesearchassistantsorconsultantsfromresearchinstitutions,universitiesorconsultantagencies.

Researchassistantsareusuallyrecentgraduateswhosupporttheinspectorinconductingtheevaluation.Theactivitiestheyperformvaryfromfilestudies,assistancewithfieldmissionsorinterviewinginformantstodraftingpartsofthefinalreport.Thewayinwhichtheyaredeployedwilldependontheirexperienceandcanchangeovertime.Researchassistantsarefullmembersoftheevaluationteam.

WiththemajorityofevaluationsIOBcontractsexternalconsultants.6Twocriteriaaredecisive:theexpertiseofexternalexpertsmustbecomplementarytothatoftheinspector,andtheymusthaveampleresearchexperience,preferablyinthecontextofevaluations.Variousmodalitiesarepossiblefordeployingexperiencedexperts.Oneormoreexpertscanbedeployedasadirectcounterpartfortheinspectorandadivisionoftasksagreedbetweenthem.Anotheralternativeisforwholeorpartoftheevaluationtobecontractedouttoaresearcheroraresearchbureau.Thewayinwhichexternalexpertiseisdeployeddependsonthenatureoftheservicesrequiredandassociatedcosts.Inthecaseofjointevaluationswithotherdonors,thewholeevaluationisgenerallycontractedout.

6.5 Cooperation with researchers in partner countries

InconductingevaluationsIOBaims,wherepossible,toworktogetherwithresearchersfrompartnercountries.Thathelpstobuildindependentresearchcapacityinthesecountries.Furthermore,researchersfrompartnercountriescanbeinvaluableastheyareoftenwellinformedonlocaldevelopmentsandrelations.However,notallevaluationsareofcoursesuitedtosuchcooperation.Thepossibilityofmakinguseofresearchersfrompartnercountriesdependsinthefirstplaceonthesubjectoftheevaluation.Somesubjectsdonotwarrantsuchachoice,forexample,becausetheevaluationwilladdressonlyDutchactivitiesorinterventionsofaglobalnature.Wherethedeploymentofexpertsisalogicaloption(forexample,evaluationofsectorpolicyinaspecificcountry)IOBactivelyexploresthepossibilities.

6 Contracting out and entering into contractual relationships are determined by the procedures laid

down in the Operational Procedures Manual (HBBZ). They will not be considered further in this document.

33

Evaluation policy and guidelines for evaluations

6.6 Methodology

Evaluationisaformofresearchthatentailsajudgment.Thecustomarymethodologyislargelyderivedfromthesocialandeconomicsciences.TheTermsofReferencegiveageneraloutlineofthemethodologytobeappliedintheevaluation.Whentheevaluationistobecontractedouttoexternalresearchers,theyareusuallyaskedtoelaboratethemethodologytheywilluseinaninceptionreport.

InaccordancewiththeOECD/DACEvaluationQualityStandards,IOBrequiresthatasoundmethodologyisappliedandthatitisaccountedforinthefinalreport.Themethodologicalchoicesmadeinanevaluationaredeterminedbytheextenttowhichtheyleadtothecollectionofreliabledataandprovideabasisforreachingvalidandreliablejudgments.Themethodsandtechniquesusedcanvaryperevaluation.

Thevalidityofanevaluationisdeterminedbytheextenttowhichthedesignoftheevaluation,themethodofdatacollectionandtheanalysisofthedataleadtoreplicableanswerstothequestionsposedintheTermsofReference.Validityrelatesto:

• thestatementoftheproblem(theclaritywithwhichtheproblemisstated,theformulationoftheresearchquestionsandthedefinitionoftheevaluationcriteria);

• theevaluationobject(thedefinition,operationandlimitationoftheevaluationobjectandsituatingitinthepolicyandinstitutionalcontext);

• thepolicytheory(thepresentationorreconstructionoftheinterventionlogicandresultlevels,andtheoperationalisationofresultmeasurementonthebasisofindicators);

• theanalysis(datasources,datacollectionandtheprocessingandargumentationofconclusionswithfindings).

Thereliabilityofanevaluationisdeterminedbytheextenttowhichtheresultsarestable,i.e.thatrepeatingtheevaluationwillproducethesameresults.Reliabilityisrelatedto:

• theresearchmethods(specificationofandaccountingfortheresearchmethodsused,verificationofdataandtriangulation);

• thescope(representativenessofthesampleandselectionofcasestudiesandspecificationofthelimitsoftheevaluation);

• independence(ofthesourcematerialandoftheresearchersinrelationtostakeholders);

• theprogressoftheevaluationandqualitycontrol(reportingontheprogressoftheevaluationandorganisedqualitycontrolthroughinternalandexternalmonitoring).

ToguaranteethevalidityandreliabilityofevaluationsasmuchaspossibleIOBchoosesacombinationofresearchmethods(‘mixedmethods’).Thiscombinationrelatestothedesignoftheevaluationandtoqualitativeandquantitativemethodsandtechniquesofdatacollectionandanalysis.

34

6 The evaluation process

The design of the evaluationThemajorityofIOBevaluationsaredesignedaccordingtotheguidelinesfor‘theory-based’evaluation.Thatislogicalbecausetheaimistoassesstheefficiencyandeffectivenessofpolicyimplementation.Thatrequirestakingpolicytheoryasastartingpoint.Themodeloftheory-basedevaluationisverysuitedtopolicyreviews.Theory-basedevaluationsfocusonadetaileddescriptionofwhathashappenedinthepastandwhy,andanalysingthedemonstrablecausesofthesedevelopments.Todothatinaresponsiblemanner,itisimportantinthefirstinstancetoreconstructthepolicytheory.Theevaluationthenassessesthevalidityofthepolicytheory.Themosttriedandtestedmethodofidentifyingaimsandinterventionstrategiesisthroughthelogicalframeworkortheoftenmoredetailedresultchain.Thismodeloffersaframeworkforsystematicdatacollection,identifyingindicatorsforsystematicassessment,anddeterminingtherelationshipsbetweenthedifferentresultlevels.

Analternativeforthisevaluationdesignistotakechangesinrealityasastartingpointratherthanthepolicytheory.Supportersof‘realistevaluation’advocatethiskindofapproach,whichfirstdevotesattentiontothecontextinwhichchangesoccurandthemechanismsinvolved.Onlyinthesecondinstancedoesitattempttoattributechangestospecificactivities.

Inrecentyearstherehasbeenincreasingemphasisinternationallyontheneedtoprovidestrongerevidenceoftheeffectivenessofgovernmentpolicy.Commonevaluationpractice,whichgenerallyprovidesplausibleexplanationsoftherelationshipbetweeninterventionsandtheireffects,isnolongerconsideredadequate.Thereisaneedtoidentifytheneteffectsofinterventions,ifpossibleinquantifiableterms.Thatmakesitnecessarytoconductimpactevaluations.ThedesireforharderimpactstudiesisalsoexpressedintheRPE2006,whichexplicitlydistinguishesex-postimpactstudiesasaseparatecategoryinadditiontopolicyreviews.InitsevaluationprogrammingIOBthereforeincludestheimplementationofaseriesofimpactevaluationsinappropriatepolicysectorsandthemes.Inpractice,however,theoptionsarenotunlimited.‘Randomisedcontroltrials’(RCTs),inwhicha‘treatmentgroup’iscomparedwitha‘controlgroup’andwhichineffectreproducealaboratorysituation,arenotapplicableinthegreatmajorityofcases.ThemostcommonalternativeforRCTsisaquasi-experimentalresearchdesign.AlthoughthiscloselyresemblesafullyexperimentalRCT,itrecognisesthattheresearcherdoesnothavefullcontroloverthecompositionofthetreatmentgroupandthecontextinwhichthemeasurementsaremade.Theselimitationsareaccountedforintheevaluationreport.

ForthesakeofcompletenessitshouldbemadeclearthatIOB’simpactevaluationsarealsodesignedonthebasisofatheoryofhowtheinterventionstobeevaluatedwork.Themostimportantdifferenceisthatthistheoryisnotexclusivelybasedonareconstructionofpolicy,butislargelyfuelledbyexistingscientificinsights.

35

Evaluation policy and guidelines for evaluations

Qualitative and quantitative methods of data collection Variousmethodshavebeendevelopedtomeasuretheeffectivenessofinterventions.Generallyspeakingadistinctioncanbemadebetweenqualitativeandquantitativeresearchmethods.Qualitativeresearchmethodsfocusmoreonprocessesandemphasisethebehaviourofactors.Theyareflexible,oftenbecausetheyhaveaneclecticapproach,andcanplacefindingsintheirculturalandpoliticalcontext.Aqualitativeapproachcangiveaplausibleexplanationofwhyacertaininterventionhasorhasnothadaneffectinagivencontext.Qualitativemethodsplayanimportantroleindetermininginternalvalidity.Itishowevernotalwayspossibletogeneralisethefindingsofqualitativeresearch.Quantitativeresearch,ontheotherhand,explicitlyaimstomakegeneralstatements(externalvalidity).Statisticaltechniquesmakeitpossibletoisolatetheeffectsofaninterventioninacomplexenvironment.Thismethodis,however,verydata-sensitive.Itrequiresalargequantityofreliableinformationtoallowtheformationofsamplepopulations.

Quantitativeandqualitativetechniquesarenotinterchangeable,butcomplementary.Quantitativetechniquescanprovideastatisticalbasisforamorequalitativeapproach,whileaqualitativeanalysecanexplainrelationshipsestablishedbyquantitativeresearch.Withoutatheoreticalframeworkandqualitativeresearchintothemechanismsbehindassumedrelationships,statisticalanalysishasnosubstance.InitsevaluationsIOBappliesbothquantitativeandqualitativetechniques.

Methods of data collection Thevalidityandreliabilityofanevaluationstandsorfallswiththereliabilityofthedatacollection.Wherethedataiscollecteddependsstronglyonthesubjectoftheevaluation.Whendevelopmentinterventionsarebeingevaluatedsomeofthedatawillusuallybecollectedinthepartnercountriesconcerned.Iftheyareavailable,existingdatabasescanbeused.

Themostcommonformsofdatacollectionare:

• File and literature studiesToobtainaclearpictureofthefieldofstudyitisnecessarytobecomefamiliarwiththecompleterangeofwrittensourcesavailable.Thisincludesinformationrelatingtointerventions,morepolicy-relateddocumentationandacademicliterature.Itrequiresworkingthroughthearchivesingreatdetail,preferablyusingasystematiclistofpointsforattention.Oneimportantpointforattentioninafilestudyistoassessthereliabilityoftheinformation.

• Interviews with key informants Generallyadistinctionismadebetweenstructured,semi-structuredandnon-

structuredinterviewmethods.Whichmethodisapplieddependsontheroleoftheinterviewintheresearchandthenatureoftheinformationthatisbeingsought.Especiallyintheinitialorientationphaseinterviewswillgenerallybeopentoallowasmanydifferentopinionsandviewpointstobecollectedfromtheinterviewees.

36

6 The evaluation process

• Group interviews Variouskindsofgroupinterviewsareusedinevaluations.Oneofthemost

commonformsisthefocusgroupdiscussion,inwhichtheinteractionbetweentheintervieweescanalsobeavaluablesourceofinformation.

• Direct measurement Alotofquantifiableinformationcanbeacquiredbysimplerecordingand

counting.Especiallyatthelevelofoutputs,thisisthemostlogicalapproach.Directmeasurementisparticularlyusefulifdataisavailablefromabaselinestudy.

• Direct observation Observationmethodscanbelimitedtothephysicaloutputsofanactivity,butcan

alsorelatetocertainprocesses.Observationtechniques(withorwithoutthedeploymentofaudiovisualaids)havealwaysplayedanimportantroleinfieldresearch.Itshouldbeaddedthattheycanbeverytime-intensive.Evaluationsgenerallymakeuseofaquick(moreorlessstructured)tourofphysicaloutputs,a‘transect’,and/orattendingoneormoremeetingsofthetargetgroupsconcerned.

• Surveys Surveysaimtocollectalargequantityofstandardiseddatafromalargepopulation.

Suchdataisverysuitableforstatisticalanalysis.Surveysmakeuseofpreviouslydrawnupquestionnaireswhicharepreferablytestedfirstandwhichcanbeinwrittenorverbalform.Theyareessentialindeterminingthestartingsituation(baseline)andareindispensableinimpactevaluations.Becausesurveyscanbetime-consumingandcostly,impactevaluationsoftenusetheresultsofpreviousquestionnairesormodulesaddedtopendingsurveys,sothattheycanservemultiplepurposes.

• Case studies Casestudiesarein-depthstudiesofasmallnumberofspecificinterventions.They

offerthepossibilityofconsideringtheperceptionsofstakeholdersorreconstructinghistories.Casestudiesareavaluablesupplementtobroadersurveys,becausetheyprovideanopportunitytoexamineandexplainrelationshipsthatbecomeapparentfromstatisticalanalysisingreaterdetail.Ifanevaluationisbasedexclusivelyonaseriesofcasestudies,itisessentialtodeterminetowhatextenttheselectedcasesarerepresentative.

37

Evaluation policy and guidelines for evaluations

6.7 Reporting

ThefinalreportofanevaluationisapprovedbytheIOBDirector,aftershe/heissatisfiedthatthecommentsofthereferencegroup,internalpeerreviewersandtheIOBDepartmentalMeetinghavebeenincorporatedadequately.IOBthensendsthereporttotheministerorministersconcernedtobeforwardedtoparliament.Onrequest,IOBcanalsoprovidetheministerwithadditionalinformationonthefindingsoftheevaluation.Ifanevaluationisaddressingapoliticallyverysensitiveissue,IOBwillinformtheministerconcernedingoodtimeontheprogressoftheevaluationand,asfaraspossible,theexpectedoutcomes.

Inprinciple,allIOBevaluationreportsareaccessibletothepublic.Thisrulecanonlybedeviatedfrominexceptionalcases.Anevaluationmight,forexample,beveryconfidentialinnatureormayonlyserveaninternallearningfunction.Thedecisionnottopubliciseanevaluationreportmustalwaysbetakenbeforetheevaluationstarts.

Thereportshouldtakeaccountofthefactthatthefindingsoftheevaluationaretobepresentedtodifferentstakeholderswithvariousinterests.ThebasicprincipleisthatIOBreportsaredesignedtoaccounttobothparliamentandsociety,andtofulfilalearningfunction.Thatsetshighstandardsforthereporting.Parliamentbenefitsfromashortandconcisepresentationofthemainfindingsandconclusions.Accountingfortheresultsandthedeploymentofthepublicfundsrequiredtoachievethemhaspriority.Policydepartmentswillbeinterestedintheextenttowhichthereportprovidesinformationthatoffersopportunitiestoadjustpolicyformulationorimplementation.Inallcasesitisimportanttoaimforatransparentpresentationoffindings,analysisandassessment.

IOBevaluationsidentifyanumberofburningissuesthat,inthelightoftheevaluationfindings,requireattention.IOBthereforeavoidsmakingconcreterecommendations,puttingitselfinthepositionofthepolicymakerandtherebyofabutcherinspectingthequalityofhisownmeat.Furthermore,IOBworksaccordingtotheOECD/DACdefinitionofevaluation,whichrefersonlyto‘lessonslearnt’,andtheguidelinesforpolicyreviewintheRPE2006.

IntheinterestoftransparencytheTermsofReferenceshouldalwaysbeattachedtotheevaluationreportasanannex.Theresearchquestionsshouldbesystematicallyaddressedinthereport,alsoifsomeofthemcannotbeanswered.Ifnecessary,additionalinformationshouldbeprovidedintheannexesontheresearchmethodsemployedandanyproblemsthatariseduringimplementationoftheevaluation.

IftherearefundamentalandsubstantiateddifferencesofopinionbetweenIOBandthestakeholdersregardingthemainconclusionsofthereport,theseshouldbeexplicitlymentionedinthereport.ThedecisiontothiswillbetakenbytheIOBDirector.

38

6 The evaluation process

Manyevaluationreportsareasynthesisofsubsidiarystudies.Especiallywherethesesubsidiarystudiescanalsobereadindependentlyitisimportanttomakethemaccessibleforpolicydepartmentsorotherstakeholders.Theyoftencontaindetailedinformationthatisnotincludedinthemainreport.TheevaluationreportsarepublishedbothinprintformandontheMinistryofForeignAffairswebsite.Subsidiarystudiesarealwaysavailableonthewebsiteandareinsomecasesalsopublishedinprintform.Inthecaseofveryextensiveevaluations,intheinterestsofaccessibility,IOBproducesasummarisedversionofthereport.BoththesummaryandthefulltextaresubmittedforinclusioninOECD/DACnetworkforevaluationservices’evaluationinventory.Completedimpactevaluationsarealsoenteredinthedatabasesetupby3IEandNONIEforevaluationsofthiskind.

39

Evaluation policy and guidelines for evaluations

7Quality control

41

Evaluation policy and guidelines for evaluations

TheIOBinspectorisresponsibleforconductingindependentevaluations.Thequalityoftheworkcarriedoutis,however,aconcernforIOBasawhole.TheIOBmanagementhasitsownresponsibilityinthiscontext,asitapprovestheevaluationreports.Toprepareforthisdecision-makingprocess,IOBhasaqualitycontrolsystemcomprisingfiveinstruments:internalpeerreviewers,referencegroups,thedepartmentalmeeting,theadvisorypanelandtheself-evaluation.

7.1 Internal peer review

Qualitycontrolbenefitsfromintensivecooperationandasystematicintercollegialmonitoring.IOBthereforehasasystemthatensuresthatinspectorscanrelyonsubstantivesupportfromcolleagueswhenconductingevaluations.Start-upreports,TermsofReference,subsidiarystudiesandthedraftfinalreportarediscussedwithtwointernalpeerreviewers,fellowinspectors,beforebeingsubmittedtothedepartmentalmeeting.Theinspectorchooses,inconsultationwiththeIOBDirector,whichcolleaguesaretoactasinternalpeerreviewers.TheDirectorappointsinternalpeerreviewersperevaluation.Theyareselectedandappointedonthebasisofthefollowingcriteria:

• coachingoftheinspector;• internalreadershipasalearningfunctionforrecentlyappointedinspectors;• evaluationexperience;• affinitywiththeevaluationsubjectandexpertiseinthethemeconcerned;• avoidingconflictsofinterest;• anequaldistributionoftheworkloadwithinIOB.

Notallcriteriaareequallyimportantandaspecificcriterioncanbedecisive.

Thetaskoftheinternalpeerrevieweristogiveadvicetotheinspectors.Thisadviceisnotbindingbutinspectorsdohavetoindicatewhattheydowithit.LastingandfundamentaldifferencesofopinionmustbesolvedinconsultationwiththeIOBDirector.

Inspectorsmustincludethecommentsandadviceofinternalpeerreviewersandashortreportoftheirdiscussionswiththereadersinthefileontheevaluation.

Theinternalpeerreviewersarenotpartofthereferencegroupfortheevaluation.

AstheIOBDirectorisresponsibleforapprovingevaluationproductshe/shecannotbeaninternalpeerreviewer.

ThedecisionwhethertomakeuseofinternalpeerreviewintheeventofjointevaluationsbetweenIOBandtheevaluationservicesofotherdonorsislefttotheIOBinspectorresponsiblefortheevaluation,inconsultationwiththeIOBDirector.

42

7 Quality control

7.2 Reference groups

AsecondformofqualitycontroloverIOBevaluationsisreferencegroups.Thesegroups,whicharecreatedforeachevaluation,arepartlyexternal,consistingofrepresentativesofthepolicydepartmentsinvolvedintheevaluation,organisationsand/orlocalbodies,andexternalexperts.AreferencegroupischairedbytheIOBmanagementandmeetsatkeymomentsduringtheprocess(e.g.todiscussthedraftToR,partialreportsandthedraftfinalreport).Theinspectorisresponsibleforreportingonreferencegroupdiscussionsand/orprovidingsummariesofwrittencommentsbyreferencegroupmembers.

Forevaluationsthatentailintensivecooperationwithresearchersinaspecificpartnercountry,asoundingboardgroupissetupinthecountryconcerned.Theformthegrouptakeswillvaryfromcasetocase.

ForjointevaluationswithotherdonorsaDutchreferencegroupisusuallynotsetup.Thereisajointsteeringgroupcomprisingrepresentativesoftheparticipatingdonors.Inthecaseoflarge-scale,complexevaluationsasmallermanagementteamissetupfromwithinthesteeringgrouptoruntheevaluationonadailybasis.Thesteeringgroupisthenprimarilyconcernedwithqualitycontrol.Inaddition,withtheseevaluations,theresearchbureauconductingtheevaluationisexpectedtoarrangeitsownqualitycontrol.Thisisoneoftheaspectstakenintoaccountwhenassessingtenders.

7.3 Departmental meeting

TheIOBdepartmentalmeetingplaysapartinqualitycontrolattwomomentsduringtheimplementationofanevaluation:whendiscussingtheTermsofReference(orsometimestheinceptionreport)andduringdiscussionofthedraftfinalreport.TheToRarealwaysdiscussedatafulldepartmentalmeeting.BeforethismeetingitmustbecleartotheinspectorwhichcolleagueshavesubmittedcommentsontheTermsofReferenceandthesubstanceofthecomments,ingeneralterms,sothathe/shecanprepareforthediscussion.Draftfinalreportsarethendiscussedonceagaininaseparatefinalmeetingbetweentheinspectorandtheinternalpeerreviewers,chairedbytheIOBDirector,beforetheDirectorapprovesthereport.Thisfinalmeeting,whichisopentoallIOBstaffmembers,hastwopurposes:itofferstheopportunityforafinalassessmentofthequalityoftheproduct,andhelpscolleaguestolearnfromeachothers’experiences.ItalsoensuresthatthefinalproductofanindividualinspectorbecomesaproductofIOBasawhole.

43

Evaluation policy and guidelines for evaluations

7.4 Advisory panel

ToincreasetheusabilityanduseofevaluationsinthefieldofdevelopmentcooperationtheMinisterforDevelopmentCooperationmadeapledgetoparliamenttoestablishanAdvisoryPanelofprominentexternalexperts.Thepanel,withitssecretariatatIOB,willplayaroleinqualitycontrolbymonitoringtheusabilityofevaluationsandofferingadviceonrequestoronitsowninitiative.Itwillbecomeoperationalinthecourseof2009.

7.5 Self-evaluation

Everyevaluationconcludeswithacriticalreviewofhowitproceeded,withtheaimofimprovingtheapproachtofutureevaluations.Theseself-evaluationsarewrittenbytheinspector,sometimestogetherwitharesearchassistantandtheinternalpeerreviewers.TheTermsofReferencefortheevaluationdeterminetheformatoftheself-evaluation.BysystematicallygoingthroughthevariouscomponentsoftheToR,theself-evaluationprovidesaclearinsightintothesuccessfulandlesssuccessfulelementsoftheevaluation.Itshouldinanycaseaddress:(1)theextenttowhichtheresearchassignmentcouldbeimplemented,(2)anydifferencesbetweentheplannedtimepathandtheactualdurationoftheevaluation,and(3)differencesbetweenestimatedandactualcosts.Self-evaluationsarediscussedinternallyandperiodicallyatthedepartmentalmeeting.

8Communication andfeedback

45

Evaluation policy and guidelines for evaluations

Communicationandfeedbackrelatetothefeedbackofevaluationresults,withaviewtothetransparencyoftheevaluationprocess,theaccountabilityfunctionandtheintegrationoflessonslearnedinfuturepolicyformulationandimplementation.Feedbackisnotlimitedtothephasefollowingcompletionoftheevaluation.Consultationwithandfeedbacktostakeholderstakeplacethroughoutthewholeperiodofimplementation:duringthepreliminarystudy,inrelationtotheTermsofReference,andduringdiscussionofsubsidiarystudiesandthedraftfinalreport.Wherenecessary,consultationandfeedbackisalsoprovidedtothepartnercountries.Whilefeedbackduringtheevaluationisimportantatimplementationlevel,aftertheevaluationhasbeencompleteditismoreimportantatpoliticalandmoregenerallevel.

Theextenttowhichcompletedevaluationsfulfilthelearningfunctiondependsonanumberoffactors.Inthefirstplace,itisamatteroftheusabilityoftheevaluationitself.Thisisdeterminedby:

• theclaritywithwhichtheexternalaim(i.e.outsidetheevaluationitself ),for whichtheevaluationresultscanbeused,isspecified;• theclarityandcompletenesswithwhichtheessenceoftheevaluationandespecially

itsmainfindingsarepresentedintheevaluationreportandthesummary;• thecompletenesswithwhichallresearchquestionsareansweredbythe

conclusions;• theextenttowhichtheresultsoftheevaluationsuggestactionsthatliewithin thecapacityoftherecipientofthereport.

Inthesecondplacelearningfromevaluationsdependsontheextenttowhichtherecipientsandusersarereceptivetothelessonsdrawnfromthestudies.Thisisrelatedtotheinstitutionalcontextandtheorganisationalculture.IOBhasnoinfluenceonthisanditisnotIOB’sresponsibility.Itisthedirectresponsibilityofthemanagementoftheorganisation(theministryorotherorganisationsthatreceiveIOBreports).

IOBdoeshoweverhavearesponsibilitytocommunicateonevaluationreports.Ascommunicationwithpolicydepartmentsiscrucialtoincreasethelearningeffect,itiscrucialthatevaluationsarediscussedintherelevantstaffmeetings.Withregardtoexternalcommunication,IOBdrawsupacommunicationplanforeachevaluation,inconsultationwiththeministry’sInformationandCommunicationDepartment.IOBincreasestheaccessibilityofevaluationresultsandensurestheirwiderdistributionbypreparingmediapresentations,givinglecturesandinterviews,writingarticles,andorganisingconferencesandseminars.

IOBevaluationsarealsoaninstrumentforrenderingaccountabilitytoparliament.WithinthreemonthsofthereportbeingapprovedbytheIOBDirector,theministersendsittoparliament,usuallytogetherwithapolicyreaction.Thetextofthereaction,whichisusuallydraftedbythepolicydepartmentconcerned,containsindicationson

46

8 Communication and feedback

howtheresultsoftheevaluationaretobeacteduponinthelightoffuturepolicy.Inthisway,accountingtoparliamentalsocontributestothelearningprocess.

Inthecaseofevaluationsinthefieldofdevelopmentcooperationthereportsarealsopublishedinthepartnercountries,offeredtothegovernmentandmadewidelyknownthroughtheorganisationofmeetings.WithaviewtomutualaccountabilityIOBinvestssystematicallyinsuchfeedbacktothepartnercountries.

47

Evaluation policy and guidelines for evaluations

297 2004 Over solidariteit en professionalisering

Evaluatie van Gemeentelijke Internationale Samenwerking (1997-2001).

isbn 90-5328-341-2297 2004 On solidarity and

professionalisation Evaluation of Municipal International Co-operation (1997-2001). isbn 90-5328-342-0 298 2004 Onderzoek naar de kwaliteit van in

2002 afgeronde decentrale evaluaties

Eindrapport. isbn 90-5328-344-7299 2005 Een uitgebreid Europabeleid Evaluatie van het Nederlands beleid

inzake de toetreding van Midden-Europese landen tot de Europese Unie 1997-2003

isbn 90-5328-347-1300 2005 Aid for Trade? An Evaluation of Trade-Related

Technical Assistance isbn 90-5328-349-8301 2006 Van Projecthulp naar Sectorsteun Evaluatie van de sectorale

benadering 1998-2005 isbn 90-5328-351-x301 2006 From Project Aid towards Sector

Support An evaluation of the sector-wide

approach in Dutch bilateral aid 1998–2005.

isbn 90-5146-000-7302 2006 Evaluatie van het Nederlandse

mensenrechtenbeleid in de externe betrekkingen

isbn 90-5328-352-8

303 2006 Dutch Humanitarian Assistance An Evaluation isbn 90-5328-352-8304 2007 Evaluatie van de vernieuwing van

het Nederlandse onderzoeksbeleid 1992-2005

isbn 978-90-5328-353-0304 2007 Evaluation of the Netherlands'

Research Policy 1992-2005 (Summary)

isbn 978-90-5328-353-0305 2007 Impact Evaluation: Water Supply

and Sanitation Programmes Shinyanga Region, Tanzania 1990-2006

isbn 978-90-5328-354-7306 2007 Chatting and Playing Chess with

Policymakers Influencing policy via the Dutch

Co-Financing Programme isbn 978-90-5328-355-4307 2008 Beleidsdoorlichting seksuele en

reproductieve gezondheid en rechten en hiv/aids 2004-2006

isbn 978-90-5328-358-5 308 2008 Het Nederlandse Afrikabeleid

1998-2006 Evaluatie van de bilaterale

samenwerking isbn 978-90-5328-359-2308 2008 Het Nederlandse Afrikabeleid

1998-2006 Evaluatie van de bilaterale

samenwerking (Samenvatting) isbn 978-90-5328-359-2308 2008 The Netherlands’ Africa Policy

1998-2006 Evaluation of the bilateral cooperation (Summary) ISBN 978-90-5328-359-2

Evaluation studies published by the Policy and Operations Evaluation Department (IOB) 2004-2009 Evaluation studies published before 2004 can be found on the IOB website at www.minbuza.nl/iob

48

Evaluation studies published by the Policy and Operations Evaluation Department (iob) 2003-2008

309 2008 Het Vakbondsmede-financieringsprogramma

Een evaluatie van steun gericht op versterking van vakbonden en vakbonds- en arbeidsrechten

isbn 978-90-5328-357-8309 2008 The Netherlands Trade Union

Co-Financing Programme An evaluation of support for trade

unions and trade union and labour rights (Summary)

isbn 978-90-5328-357-8309 2008 El Programa de Cofinanciamiento

Sindical Una evaluación del apoyo orientado

al fortalecimiento de sindicatos y de derechos sindicales y laborales (Resumen)

isbn 978-90-5328-357-8310 2008 Clean and sustainable? An evaluation of the contribution of

the Clean Development Mechanism to sustainable development in host countries

isbn 978-90-5328-356-1311 2008 Impact Evaluation: Primary

Education in Uganda isbn 978-90-5328-361-5312 2008 Impact Evaluation: Primary

Education in Zambia isbn 978-90-5328-360-8313 2008 Xplore-programma isbn 978-90-5328-362-2314 2008 Primus inter pares Een evaluatie van het Nederlands

EU-voorzitterschap 2004 isbn 978-90-5328-364-6315 2008 Impact Evaluation: Support to Rural

Water Supply and Sanitation in Dhamar and Hodeidah Governora-tes, Republic of Yemen

isbn 978-90-5328-364-6316 2008 Be our guests Beleidsdoorlichting Nederland als

gastland van internationale organisaties en Samenvatting

isbn 978-90-5328-370-7316 2008 Be our guests Policy review on hosting international

organisations in the Netherlands and Summary

isbn 978-90-5328-371-4

316 2008 Be our guests Examen de la politique meneé par les

Pays-Bas en tant qu'État hôte des organisations internationales (Sommaire)

isbn 978-90-5328-372-1317 2008 Sectorsteun in milieu en water isbn 978-90-5328-369-1318 2008 Samenwerking met Clingendael Evaluatie van de subsidieovereen-

komst tussen de ministeries van Buitenlandse Zaken en Defensie en

Instituut Clingendael isbn 978-90-5328-365-7319 2008 Meer dan een dak Evaluatie van het Nederlands beleid

voor stedelijke armoedebestrijding isbn 978-90-5328-365-3320 2008 Het tropisch regenwoud in het

OS-beleid 1999-2005 isbn 978-90-5328-374-8321 2009 Maatgesneden Monotoring 'Het

verhaal achter de cijfers' Beperkte beleidsdoorlichting

Medefinancieringsstelsel 2007-2010 isbn 978-90-5328-374-5322 2009 Draagvlakonderzoek Evalueerbaarheid en resultaten isbn 978-90-5328-375-2323 2009 Preparing the ground for a safer

world Evaluation of the Dutch efforts to control landmines and explosive remnants of war 1996-2006

isbn 978-90-5328-377-6

Evaluation policy and guidelines for evaluations

49

50

51

Evaluatiebeleid en richtlijnen voor evaluaties

52 Publishedby:

MinistryofForeignAffairsPOBox20061|2500EBTheHague|TheNetherlandswww.minbuza.nl

Layout:Eindeloos,TheHaguePrinter:OBT,TheHague

©MinistryofForeignAffairs|October2009|Ordercode:BZDR6662/E

ratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouP

etersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAthairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampala

azLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroe

nouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSa

aticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaK

orkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujum

houGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.P

aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMia

ortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotono

dIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiT

aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MpalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskou

ortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisVelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgH

openhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauW

restBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/K

etersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntweenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKa

azLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBa

asChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKiazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSan

alle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaK

MumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetB

furt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoP

aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaMan

ekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotono

adHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyT

aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonoJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskou

ortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhou

openhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusW

BernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewY

auloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWtislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIst

azLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouPetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAthe

iroCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeR

okioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroenouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaK

MumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetB

aDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoP

enenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaP

obeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColombo

elsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiaortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyT

aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonodIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskou

ortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTjnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavanna

aoundéZagrebHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisV

elgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewY

auloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWrestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStad

azLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwe

enosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsP

eheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBaasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKi

azLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSanalle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetB

alaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoP

ashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitK

obeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCa

HanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.P

aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonoadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideo

ortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotono

JubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioT

aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskou

ortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHel

azLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonW

tislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouP

etersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheiroCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampala

azLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroe

nouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSa

aticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaK

orkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujum

houGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.P

aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColombo

IOB

Evaluation policy and guidelines for evaluations

Published by:

Ministry of Foreign Affairs PO Box 20061 | 2500 EB The Hague | The Netherlandswww.minbuza.nl© Ministry of Foreign Affairs| October 2009 | Order code: BZDR6662/E

olicy and Operations Evaluation Department | IOB | October 2009 | Policy and Operations Evaluation Department | IOB | October 2009 | Policy and Operations Evaluation Department

This document describes the evaluation policy and guidelines for evaluations of the Policy and Operations Evaluation Department (IOB) of the Dutch Ministry of Foreign Affairs. In publishing these guidelines, IOB wishes to provide transparency to third parties: colleagues at the Ministry of Foreign Affairs, the Dutch parliament, organisations and institutions affected by IOB evaluations, users of the evaluation results, and interested citizens. IOB will periodically review its policy and the guidelines and, if necessary, revise them.