evaluation of the unicef india summer internship

EVALUATION OF THE UNICEF INDIA SUMMER INTERNSHIP PROGRAM 2008-2011 For UNICEF India, December, 2012

By Sonal Zaveri, PhD

2

Table of Contents Abbreviations ................................................................................................................................................ 3

Executive Summary .............................................................................................................................. 4

1. Introduction .................................................................................................................................. 10 1.1 Justification and Purpose: Evaluation of the Summer Internship Program .......................................... 10 1.2 Description: The Summer Internship Program (IP) ............................................................................... 10

2. Evaluation Methodology ........................................................................................................... 14 2.1 Evaluation Approach ............................................................................................................................. 14 2.2 Management of Evaluation ................................................................................................................... 17 2.3 Stakeholder Participation ..................................................................................................................... 18 2.4 Data Collection and Analysis ................................................................................................................. 18 2.5 Challenges ............................................................................................................................................. 20

3. Findings .......................................................................................................................................... 20 3.1 Relevance .............................................................................................................................................. 21 3.2 Effectiveness ......................................................................................................................................... 30 3.3 Efficiency ............................................................................................................................................... 36 3.4 Sustainability ......................................................................................................................................... 37 3.5. Possible Impact .................................................................................................................................... 38

4. Overall conclusions, lessons learned and recommendations ......................................... 43 Attachment One: Terms of Reference ........................................................................................................ 51 Attachment Two: Short Term Contract of host organization ..................................................................... 58 Attachment Three: Evaluation Matrix ........................................................................................................ 61 Attachment Five: Relevance of IP Program: Related Tables ....................................................................... 66 Attachment Six: Effectiveness of IP program: Related Tables .................................................................... 68 Attachment Seven: Possible impact of IP: Related tables .......................................................................... 71 Attachment Eight ........................................................................................................................................ 74 Diagram: Theory of Change Internship Program ........................................................................................ 74

3

Abbreviations

HR Human Rights

ICO India Country Office

IP Internship Program

IR Intermediate Result

KCCI Knowledge Children on Children in India

KI Key Informants

KM Knowledge Management

MDG Millennium Development Goals

NGO Non Government Organization

SPPME Social Policy Planning Monitoring and Evaluation

TOR Terms of Reference

USD US Dollar

4

EXECUTIVE SUMMARY

Context

The purpose of the evaluation is to answer the question ‘whether the IP is an appropriate approach to knowledge management for UNICEF India which continues to move upstream and is increasingly expected to play a role as a policy organization’. Started in 2005 and into its eighth year, the IP has not been evaluated and as UNICEF India moves into its next programming cycle 2013-2017, the evaluation will address whether the original objectives of the IP were met and to what extent it contributed to the knowledge management results of UNICEF India. The IP program invests about USD150,000 annually (In 2009, the program cost was USD 217,000 for about 80 students and in 2011 it was USD 142,069 for 41 students).

The India Summer Internship program was conceived as part of a broader initiative to build a Knowledge Community on Children in India (KCCI), UNICEF India’s first knowledge management (KM) efforts initiated in 2005. The importance of KM to UNICEF’s work in India has been increasingly recognized and one of the key strategies for achieving the results of the 2008-12 GOI-UNICEF Country Program was “providing technical assistance and support to improve KM systems and sharing of lessons learned.”1 The ultimate goal of KCCI is to learn lessons from the field in order to improve programming and eventually influence policies related to children.2 The KCCI includes four components: 1) Summer Internship Program 2) Publications 3) Website and 4) Resource Center. The purpose of this evaluation is to evaluate the Summer Internship Program. The current UNICEF India Five Year Country Programme is from 2008 to 2012 and the evaluation of the IP follows the same timeline i.e. from 2008-2011

In the IP program, UNICEF India partners with research and academic institutions to encourage young students and scholars from around the world to engage in development issues pertaining to India's children and women. The assignment involves a combination of desk and fieldwork at the district or village level and/or at state and union level, with interns being grouped into teams that are based in field offices and work under the supervision of selected research institutions/non-governmental organizations. Every year between 40 to100 interns Indian and international, selected through a rigorous selection process, representing over 40 countries and with a Masters or PhD in development related disciplines participate in a ten-week long IP usually in the months late May to early August. The UNICEF India Internship program is not an individual assignment; teams of four work on different assignments in different states; national and international interns are included in each team providing opportunities for learning and sharing at multiple levels – personal and professional. The internship program is a planned study with a case study (print and other media) as an expected output. Further, case studies are reviewed and only those that pass the quality criteria (adherence to TOR, good understanding of broader context, describes value added to current programming practice, process and results describe actual changes in physical condition/behavior/attitude of project stakeholders, quality and source of quantitative and qualitative evidence, strategy and implementation describes successes and potential for replication) are published and disseminated. The case studies are available in print as well as on the KCCI website.

1 Terms of Reference or TOR

2 UNICEF India Summer Internship Programme: Linking Action and Policy for Children in India, Concept

Note

5

The Social Policy, Planning, Monitoring and Evaluation (SPPME) team manage the Summer Internship Program. It includes the Knowledge Management Officer3, who spends the majority of her time (70%) managing the Internship program assisted by hired consultants during phases of the internship such as selection of interns and managing the ten-week internship program. A Research and Evaluation Specialist who provides 10% of her time assists the PO to supervise the program4.

The original objectives of the IP as formulated in 2005 were to: a) Produce high quality documentation of innovations in service delivery, what works

(and what does not), why, and under what conditions, so as to use this to influence and inform the development discourse and disseminate evidence based research amongst government functionaries, development practitioners and policy-makers

b) Collect field-work based data and information relating to children across India which will offer comparable snapshots into progress made in selected issues relating to children’s development, as well as complement UNICEF’s documentation needs.

c) Support and develop a cadre of young research and development professionals with interest, commitment and skills relating to children’s development in India

d) Develop partnerships with leading research institutes in India and promote greater interest in social development research.

e) Promote awareness about UNICEF activities among graduate students in India and

The evaluation was conducted from May to September 2012 for a period of 30 days. The sample was 109, including 67 interns, 15 host organizations, 16 UNICEF managers, 2 from IP secretariat, 3 from KCCI management and 7 key decision makers. Data was collected through online survey, which was piloted and pre-tested using Zoomerang, and remote interviews. The government representative was not able to confirm an interview appointment.

Findings

The report presents findings according to relevance, efficiency, effectiveness, sustainability and possible impact.

1) Relevance: The IP program is to a great extent relevant to and able to respond to the KM needs as outlined in the various documents. The IP program contributed a small but important niche towards achieving India’s KM results especially in knowledge generation and to some extent in knowledge use. There is a mismatch regarding UNICEF and host organizations’ perception of the relevance of different IP objectives reflecting a lack of coherence about IP. The IP partially contributed to the knowledge generation of other IRs; however it is difficult to track use. There is a rigorous process of selecting the best interns, nationally and internationally, use of criteria to ensure quality of the KM product (intern’s case study) and strategy for dissemination (print and web). Use is dependent on many factors of which knowledge generation is just one. The IP relevance to the new country program is clear in terms of its contribution to knowledge management on issues of importance to children. In terms of the structure and design, the IP is eminently suited for KM but its design and management to a small extent only encourages use.

3 The Knowledge Management Officer manages the day to day work of the Internship Program and, for

the purpose of this evaluation, is referred to as the IP Program Manager. 4 Based on interview with IP Program Manager, Amrita Singh

6

2) Effectiveness: There were limitations in the ability of the host institutions to support the interns in the knowledge and skills needed for the development of the case study. Host organizations need to be selected according to their research capacity, content knowledge and ability to have a committed mentor for the interns. The host should facilitate and guide and provide the space for interns to complete their research. The quality of the case study was dependent on the clarity of the TOR, strength of the research design and timely support of both UNICEF state offices as well as the host. Limited participation of government and host organization in developing the TOR was seen as a gap. UNICEF’s regular follow up during data collection and feedback on interns’ draft case study was considered important to ensure quality and additional quality criteria were identified. Streamlining host-UNICEF-intern relationship regarding management of interns’ study, clear and consistent communication about study objectives amongst all stakeholders, use of knowledge product and providing the space to interns to independently express their findings were important for the effectiveness of the IP. Dissemination and advocacy for use of case study has not targeted key decision makers, the government or community.

3) Efficiency: The IP program may be considered as efficient as the cost of implementing the IP program has remained more or less consistent over the years. Stakeholders such as interns and host organizations contribute to the IP resources. The interns particularly the international interns fund their own travel to India. The host organizations receive modest support to house the interns and for related expenses of fieldwork. Case studies, a KM product, are cost effective considering the short time and skill available to produce them by a team of young professionals using original field data.

4) Sustainability: The IP program is sustainable since its systems are in place and has involved a variety of stakeholders at different levels, country, state and local. The IP program of UNICEF ICO has a unique design of cross-cultural teams working at grassroots to produce a knowledge product – however, there is little evidence regarding how the case studies were used. UNICEF ICO can continue the program as is. The evaluation could not explore if the program could be ’adopted’ by others since these decisions are dependent on many other factors. Currently, some parts of the IP are outsourced but fully outsourcing selection of interns, teams, matching with organizations, and reviewing case studies requires dedicated staff and expertise wherever it is housed. The evaluator can only opine that maintaining quality standards is important and that the outsourcing must weigh efficiency considerations with quality maintenance. Whether government can take it over is a moot point since the many pieces that make the IP program work are unique and difficult to replicate to the quality and flexibility needed in a bureaucratic environment.

5) Possible Impact: Overall, the IP program has contributed to interns’ awareness of development issues with field experience, ability to develop a knowledge product such as the case study using primary data collection and unexpectedly learned how to work in multi-cultural and multi-disciplinary teams. The case studies were disseminated but lacked an explicit strategy for use. Expecting that interns’ case studies can influence policy is unrealistic even when the KM products are vetted for quality. Overwhelmingly, the interns, host organizations and UNICEF have suggested that the benefits of the IP justify its continuation.

Conclusions and Recommendations

The IP program is relevant with knowledge management strategies and has by and large fulfilled the objectives of the IP program. Due to the effective management and structured design of the IP regarding linking a KM result with developing a cadre of young professionals, there has been a continuous demand for internship over the years even though the numbers accepted have decreased. Also the number of products – case studies and videos (in earlier

7

years) has steadily increased with acceptance for publishing also going up. The IP program promoted multi-cultural and multi-disciplinary understanding and resonated with the international mandate of UNICEF.

Systematic, transparent and substantive communication has emerged as a key factor for the successful implementation of the IP. This includes the importance for regular communication a) between UNICEF state offices, government and host organization not just about logistics but of more substantive issues b) between UNICEF state offices, host organizations and government to select the most appropriate areas of study considering the two month period available c) between UNICEF ICO and the state offices regarding quality of research and guidelines for production of a quality product by interns and not just about contracting host organizations or managing logistics d) between interns and UNICEF enabling transparent reporting of findings and how they can be represented in the case study e) between UNICEF ICO, state offices, sector specialists and leadership for the more substantive issues of the IP such as how to use the product (also depends on what is selected for research and its quality) and finally f) communication with interns both in the short term before their arrival ( and connecting with other interns past and present, proposed state offices and host organizations); immediately after the internship to communicate the status of publication (or not); information regarding use of the case study; and in the long term after they move on to catalyze on their India experience. Interns’ keen desire to continue to contribute to the IP program needs to be actively and strategically exploited – having a Facebook page is only one step in the process.

UNICEF ICO, state offices, host organizations, government and interns’ collaboration and shared expectations affected the case study quality and use. If any of these stakeholders have different expectations, then it is impossible for the interns to produce a case study fit for publication. Managing expectations was important including the host and UNICEF state offices ability to provide the interns the ‘space’ for honest data collection and reporting (and that one can learn from what does not work well) and interns for representing data in the context in which it is collected.

A network of research organizations has been created over the years and it is important to nurture this relationship as well as to continually add new organizations to the pool. The IP lesson is that in short term relationships where the benefit is perceived in only one direction (for the interns and UNICEF), the engagement is likely to be limited and be compromised in quality. To liaise with the best research organizations available – academic or otherwise – requires a long term relationship perceived to be of mutual benefit.

Impact of the IP on Interns, host organizations and UNICEF and recognition of the benefits is substantiated by the overwhelming recommendation for the program to continue. The IP has enriched interns in many ways – being better prepared to enter the development field, understanding children and development needs in India, able to network and make useful contacts and being able to apply their research skills. As one intern mentioned ‘one of the great things about the program is the creative freedom within a framework of scientific rigor’. UNICEF recognizes the role in its own knowledge building and although the request for hosting interns is now voluntary, the demand has remained constant from the UNICEF state offices indicating they do value the IP. Host organizations have added to their own knowledge building through the IP program and believe that their relationship with UNICEF is stronger. Although government is consulted and information shared, their engagement is limited for various reasons.

8

Sustainability of the IP program whether by UNICEF or others will depend on many factors including how UNICEF positions the IP as a program of value for knowledge management. It will depend, besides recommendations for improving the program, upon how much UNICEF leaders, staff at country and state level believes in the interns’ ability to conduct research, host organization’s to mentor them, government to support the IP and the overall quality of the case studies that encourages not only dissemination but a clear strategy for use.

The eleven recommendations based on lessons learned are clustered around three broad categories: strategic, programming and operational issues based on the analysis of findings according to relevance, effectiveness, efficiency, sustainability and impact. The responsibilities for the implementation of recommendations have also been mentioned. The recommendations (R) are as follows:

Related to Strategic Issues: R1) All stakeholders to strategically disseminate case studies and decision makers, especially in UNICEF, to advocate for use: Dissemination is not use and strategies are needed to create pathways for the creative use of the publications by all stakeholders including interns themselves and the academic institutions they come from. A product by interns over a two month period has to have realistic expectations; it is unlikely to influence policy (as expected in the IP objective). R2) UNICEF to revise and reorder objectives of IP to realistically reflect what it can achieve and to manage multiple expectations: Some of the objectives of the IP and their order of importance need to be reviewed. R3) UNICEF ICO and state offices to brand and advocate the IP nationally and internationally: Champions in UNICEF at leadership levels at ICO and the state would energize and bring value to the IP in India. Internationally, the unique design of the IP – the selection process, multi-cultural teams of interns, the collaboration with research organizations at state level, the field work opportunity and a concrete KM product vetted for quality – needs be branded and disseminated

Related to Programming Issues: R4) UNICEF to support high quality research skills and provide timely technical backstopping: Research capability and inputs by all stakeholders in the IP is important – interns, UNICEF ICO, state offices and host organization – to produce a quality document. This may mean additional research skill building workshops or recruiting personnel with research skills. R5) UNICEF to extend the reach of IP recruitment in India and abroad: The selection of interns has been streamlined and outsourced to a consultant that ensures transparency of selection from a variety of applicants from India and abroad. However, catchment areas from India and internationally have remained the same over the years for various reasons. R6) All stakeholders to make creative use of the short timeline through early introductions, understanding the context and guidelines for quality documentation: The time period of two months is very short for interns to understand and adjust to the context in India so the process can start early and will mean tweaking of the logistics. R7) UNICEF to collaborate with key stakeholders to develop good quality TORs: The research topic must be an expressed need of relevant stakeholders especially government and well articulated so that government, host organizations, interns and UNICEF clearly understand what data is to be collected, why and how it is to be used.

Related to Operational Issues: R8) UNICEF ICO and state offices to be proactive in identifying and building relationships with quality research institutes to support the IP as well as other research needs: Selection and motivation of quality host institution is key to the success of the IP as they will be the mentors for interns and their first point of contact. R9) UNICEF ICO and states to develop/strengthen strategies for advocacy and use of case studies (and other knowledge products) with government: The government is informed and supports field collection, but they need to be actively involved in the TOR development and findings need to

9

be disseminated to them in a succinct manner. Only then is use possible. R10) UNICEF ICO to strengthen IP ownership by UNICEF and other stakeholders at state level: The IP is currently viewed as a UNICEF ICO program driven by SPPME, although participation of state UNICEF offices is voluntary and the IP is decentralized in operation. R11) UNICEF to actively build alumni networks: UNICEF has not engaged actively in the over 400 alumni network created over many years missing out an opportunity to build knowledge and partnership networks and advocates

10

1. INTRODUCTION

1.1 Justification and Purpose: Evaluation of the Summer Internship Program

The purpose of the evaluation is to answer the question ‘whether the IP is an appropriate approach to knowledge management for UNICEF India which continues to move upstream and is increasingly expected to play a role as a policy organization’. Although into its eighth year, the IP has not been evaluated and as UNICEF India moves into its next programming cycle 2013-2017, the evaluation will address whether the original objectives of the IP were met and to what extent it contributed to the knowledge management results of UNICEF India. The IP program invests about USD150,000 annually (In 2009, the program cost was USD 217,000 for about 80 students and in 2011 it was USD 142,069 for 41 students).

The IP program began in 2005 and has continued to date. Assessments completed by interns every year have consistently indicated that over 90% of the past participants were satisfied with the program. Anecdotal evidence suggests that it has made progress towards the objectives – for example, past interns report working in areas related to children’s development, studies by interns have been reported to be used to increase the knowledge base and for advocacy. A Facebook group consists of 325 friends and the Knowledge Community for Children in India (KCCI) website www.kcci.org.in features the case studies that make it to publication. The IP program is one component of the KCCI.

The current UNICEF India Five Year Country Programme is from 2008 to 2012 and the evaluation of the IP follows the same timeline i.e. from 2008-2011. The timeline for evaluation was defined in order to validate relevance of the IP to the current Country Programme strategic direction.

This report presents the findings that will answer the following: Provide credible and reliable evaluation of what the program achieved in relation to program design, implementation, impact on interns, host organizations and overall results for UNICEF knowledge building.

1.2 Description: The Summer Internship Program (IP)

1.2.1 Background: Knowledge Community for Children in India (KCCI)

The Knowledge Community for Children in India (KCCI) efforts initiated in 2005 was UNICEF India’s first knowledge management effort. UNICEF India did not adopt a KM strategy per se but drafted the KCCI Turning Policy into Practice concept paper in 2005.5

What is knowledge management?

Definition used in the UN System “An integrated, systematic approach to identifying, managing and sharing an organization’s knowledge, and enabling groups of people to create new knowledge collectively in order to achieve the objectives of the organization” Definition used by UNICEF HQ “a management activity that seeks to enhance the organization, integration, sharing and delivery of knowledge. There are three major elements of knowledge management: knowledge, tools and systems. 5 Report of Information and Knowledge Management: Learning from KM Experiences, UNICEF ROSA,

April 2008. Pg. 54

http://www.kcci.org.in/

11

Definition (working definition) used by UNICEF ROSA “management of information and research in order to improve the lives of children of South Asia, in line with the MDG commitments”. Source: Learning from KM Experiences, Pg5

The importance of KM to UNICEF’s work in India has been increasingly recognized, and one of the key strategies for achieving the results of the 2008-12 GOI-UNICEF Country Program was “providing technical assistance and support to improve KM systems and sharing of lessons learned.” 6

UNICEF India will enter into a new five-year Country Programme from 2013. Knowledge management will remain as one of the key strategies and as an Intermediate Result ‘Evidence-based strategic knowledge and advocacy tools on prioritized issues related to child rights developed, documented and disseminated’ contributing to the programme component result 4: Policies, practices, programmes, public opinion and social norms advance the rights of children, adolescents and women.

The aim of the KCCI initiative in India is to ensure that relevant knowledge and leading research is brought to bear on policies and programs for children in the country and to that end, promotes information sharing. In partnership with the GOI, UNICEF addresses gaps in knowledge regarding implementing successful programs for children, answering questions of what worked and what could be improved. The ultimate goal of KCCI is to learn lessons from the field in order to improve programming and eventually influence policies related to children.7 The KCCI includes four components: 1) Summer Internship Program 2) Publications 3) Website and 4) Resource Center. The purpose of this evaluation is to evaluate the Summer Internship Program.

1.2.2 Context: Internship Program

UNICEF offers internship program at both headquarters and at country level. The India Summer Internship program was conceived as part of a broader initiative to build a Knowledge Community on Children in India (KCCI), UNICEF India partners with research and academic institutions to encourage young students and scholars from around the world to engage in development issues pertaining to India's children and women. The internship offers an opportunity to explore issues related to child rights in India across a variety of areas: child protection, health, nutrition, education, HIV/AIDS, and water and sanitation. The assignment involves a combination of desk and fieldwork at the district or village level, with interns being grouped into teams that are based in field offices and work under the supervision of selected research institutions/non-governmental organizations.

IP is advertised on various websites: UNICEF India and Global site; DevNetJobsIndia and DevNet (Global); KCCI website. It is also sent to a pre-identified list of universities across the world, the alumni network and research institutions who have hosted interns in the past.

6 Terms of Reference or TOR

7 UNICEF India Summer Internship Programme: Linking Action and Policy for Children in India, Concept

Note

12

Applications are invited for the year’s Programme, until January. All the applications go through a competitive selection process and selected interns are notified by 15th of March. Equal weightage is given to a) Applicant's academic and research training b) Originality, quality and innovativeness of Statement of Purpose c) Knowledge of written English d) Demonstration of a commitment to development either through previous study/ publication/ relevant work experience.

Applicants should be enrolled in a Masters-level graduate programme or a higher degree program, in one of the following disciplines: anthropology, child psychology, demography, economics, education, engineering, human rights, international development, journalism, legal studies, public health, sociology, statistics, rural development, social work, or any other relevant area. Those who have completed their Masters level graduate programmes the year before are also eligible to apply. Applicants are expected to work in a multi-cultural environment and establish harmonious and effective working relationships with individuals from different backgrounds. Applicants are expected to have excellent command of English, both written and spoken.

All interns are expected to cover costs such as travel from home to duty station and back, as well as accommodation and food for the entire duration of the internship programme. However, UNICEF provides a modest Rs. 20,000 as partial support to all interns for the two-month period. UNICEF India also meets the expenses incurred on the following: (a) travel from Delhi to the assigned state capital after the orientation workshop, (b) field trips from the state capital to the field and, (c) travel from state capital to Delhi for the closing workshop at the end of the internship programme.

The UNICEF India Internship program is not an individual assignment; teams of four work on different assignments in different states; national and international interns are included in each team providing opportunities for learning and sharing at multiple levels – personal and professional. The IP Secretariat selects members for each team and matches them with the research organization and the topic of study, attempting the best match possible between the interns’ interests and requests from the UNICEF state offices. The internship program is a planned study and includes a case study (print and other media) as an expected output. Further, case studies are reviewed for quality and only those that pass the criteria are published and disseminated. The case studies are available in print as well as on the KCCI website.

1.2.3 Summer Internship Program Objective

Every year between 40 to100 interns participate in a ten-week long IP usually in the months late May to early August. The Orientation Workshop is held in Delhi for three days where the interns meet for the first time. The meet their team members, IP secretariat, intern alumni, host organization representatives, UNICEF state managers and Indian thought leaders. The program comprises of substance and skill sessions, with the aim of providing information on issues and challenges surrounding policy, administrative structures and political economy. There are also sectoral introductions to child development and nutrition, child rights, HIV, SPPME, decentralized planning, community participation and emergency. The interns travel to the states for about 8 weeks and then return to Delhi for the presentation of the case studies and feedback on the IP experience. Case studies are usually focused on work supported by UNICEF in different states such as child development and nutrition, child rights, HIV, SPPME, decentralized planning, community participation, and emergency.

13

Interns from India and abroad work in collaboration with leading research institutes or implementing partners, combining their energy and fresh perspectives with the experience and expertise of research organizations towards generating strong evidence based qualitative and quantitative research that results in a case study8. The expected output from each team is a 30-page lessons learned/good practice paper on identified intervention along a pre-determined framework.

Interns are grouped in teams of four; two are national interns and the other two, international. The research organizations/implementing partners called partner or host organizations house and support the interns in data collection and analysis in each state to develop the case studies, supported by the UNICEF state office as well as sector specialists from the UNICEF country office.9 The UNICEF state offices are directly involved in developing the TOR based on identified need for the interns’ research that culminates in the case study, in selecting the host organization, completing contracting formalities with the host organization and providing technical support to interns along with the host organization. Although UNICEF ICO manages the Internship Program, it is decentralized in implementation.

Since 2005, about 500 interns have participated, hosted by over 70 research and academic institutions. Interns have worked in all states where UNICEF has presence except Chhattisgarh. Interns from over 22 countries have documented 129 case studies out of which 88 have been published after being reviewed by the field offices and program sections.10 The number of interns has decreased over the years: 105 in 2005, 100 in 2006, 57 in 2007, 82 in 2008, 66 in 2009, 44 in 2010 and 41 in 2011.

The Social Policy, Planning, Monitoring and Evaluation (SPPME) team manage the Summer Internship Program. It includes the Knowledge Management Program Officer (PO)11 who spends the majority of her time (70%) managing the Internship program assisted by hired consultants during phases of the internship such as selection of interns and managing the ten week internship program. A Research and Evaluation Specialist, who provides 10% of her time, assists the PO in oversight of the program12.

The original objectives of the IP as formulated in 2005 were to:

f) Produce high quality documentation of innovations in service delivery, what works (and what does not), why, and under what conditions, so as to use this to influence and inform the development discourse and disseminate evidence based research amongst government functionaries, development practitioners and policy-makers

g) Collect field-work based data and information relating to children across India which will offer comparable snapshots into progress made in selected issues relating to children’s development, as well as complement UNICEF’s documentation needs.

8 Concept note

9 See Attachment Two delineating the short term agreement with host organizations.

10 TOR - See Attachment One.

11 The Knowledge Management Officer manages the day to day work of the Internship Program and, for

the purpose of this evaluation, is referred to as the IP Program Manager. 12

Based on interview with IP Program Manager, Amrita Singh

14

h) Support and develop a cadre of young research and development professionals with interest, commitment and skills relating to children’s development in India

i) Develop partnerships with leading research institutes in India and promote greater interest in social development research.

j) Promote awareness about UNICEF activities among graduate students in India and abroad

In 2008 the research focus for all interns was on equity and inclusion issues intrinsic to UNICEF’s human rights programming and excerpts of the case studies were published in a document13’Learning from Practice: Strategies for promoting equity and inclusion’. All the interns worked on one issue across states. In 2009, the overall theme was linked to capacity building. In subsequent years, interns have worked on a variety of issues usually identified by the UNICEF state offices. Typically, the shortlisted case studies (based on a qualitative review) are printed and available on the website www.kcci.org.in.

2. EVALUATION METHODOLOGY

2.1 Evaluation Approach

The terms of reference (TOR) guided the evaluation design and included the Human Rights principles of inclusion, participation and consultation. Guided by these principles, the evaluation used a mixed methodology, involving a wide range of stakeholders, developed data gathering instruments that were tailored for each of these stakeholders and ensured that the data was collected and analyzed responding to the human rights and gender equality principles as well as the DAC criteria. This included adequate samples for various stakeholder groups, triangulating the data from various sources and validating the design with reference group members.

The evaluation approach was developed through review of documents, information regarding the output and outcome indicators, discussions with the UNICEF team, meeting the 2012 interns (not part of the study sample) as well as past interns informally at the 2012 intern orientation workshop in Delhi in May 2012. The evaluation used the theory of change approach to examine the links between outcome, activities and results. Along with the theory of change, it reviewed the logic of inputs, outputs, outcomes and goals as represented in the model below based on the 2008-12 GOI-UNICEF Country Program Action Plan Results Matrix.

Following the UN Evaluation Group guidelines, the evaluation adheres to the criteria of relevance, efficiency, effectiveness, sustainability and impact. The evaluator engaged with multiple stakeholders including interns to capture what worked, what were the challenges and lessons learned following ethical guidelines. Participation in the survey and interviews was voluntary and random methods of selection were used to ensure representation without bias. The evaluator had an opportunity to observe first hand the opening and orientation workshop for the 2012 interns. Informal interaction with some of the interns helped the evaluator to better

13

https://docs.google.com/viewer?a=v&q=cache:7UkwWw5Bd30J:www.unicef.org/india/Booklet_%28High-res_for_Print%29.pdf+&hl=en&gl=in&pid=bl&srcid=ADGEEShbUIC75DCH-RuVu5_y6RPt_ZfAYOLUunZDdmAkXrpJZVGmyPj7jC5DQgTGM5rnOAVqzhVV9RpgHwRW-an8LnfhGvVWp0QmRpxFIdNkdz44ubS78w_tfGs6iTb2Pcc3zZUa8jtO&sig=AHIEtbQom1yP6vcxEvZ2Mg_U8IJ4ClR97g


15

understand the program and design the evaluation questions. The one-month evaluation was conducted between June to September 2012. The findings will recommend the future plan and design of the IP as well as refine the KM strategy for the new country program

16

Diagram One: Logic Model for IP

Guided by this logical model, the evaluation assessed whether the change happened. It assessed the degree to which the inputs from the various sources and activities conducted produced the stated outputs. It further assessed whether the four outputs resulted in the intermediate outcome of internship program continuing to produce studies on topical areas and finally it assessed whether it lead to the outcome of ‘Accessible and user friendly user base in the public domain’ and whether that led to the overall goal of knowledge management. A set of questions that measured the standard evaluation criteria of relevance, effectiveness, efficiency, sustainability and impact followed the human rights and gender perspectives and were shared and validated by the Evaluation Management Group through the inception report prior to the commencement of the evaluation. An evaluation matrix was prepared that addressed the

IP Output 1.Host institutions have knowledge and skills to support interns in development of case studies

IP Output 2: Intern teams have the commitment and expertise to understand situation, collect data, analyze and write up quality case studies

IP Output 3: Duty bearers at state level understand need for and use case studies for program and policy.

IP Output 4: UNICEF at center and region are able to support interns, ensure useable quality products and advocate for use.

Set of activities related to output 1




Inputs from UNICEF ICO and sectoral specialists

Inputs from SPPME unit & IP secretariat

Inputs from govt. and related agencies

Inputs from interns

Inputs from UNICEF state offices

Inputs from host organization

Goal: (Knowledge Management Result reformulated in 2011): Strategic knowledge on MDGs, social

inclusion, equity and child rights developed documented and disseminated to inform programming and policies for multiple audiences.

Outcome: Accessible and user friendly knowledge base on MDGs/children’s issues (especially on social exclusion issues) available in public domain for diverse users

Intermediate outcome: (one of many) Continued internship program to undertake topical study or research (in the case of IP, case studies)

17

questions in each criteria and aligned them with the outcomes, outputs, activities, indicators and data sources providing a logical flow to the evaluation. (Please see Attachment Three)

2.2 Management of Evaluation

The SPPME section under which the KCCI and in particular the Summer Internship Program is housed commissioned this evaluation. The IP Program Manager for the Internship Program located in UNICEF Delhi, Ms. Amrita Singh was the point person for the evaluation process along with the Research and Evaluation Specialist Ms. Michiru Sugi. The head of the SPPME unit provided an overview of the evaluation needs. The IP Program Manager coordinated the supply of relevant documentation, arranging appointments, reviewing draft documents, coordination with the reference group and the timely dissemination and feedback collation for the independent consultant’s deliverables. The independent consultant, Dr. Sonal Zaveri was selected through a competitive process and started work on May 27th, 2012.

2.2.1 Evaluation Management Group

The Evaluation Management Group was involved in the selection of the consultant, the review of the TOR and was aware of the program and evaluation objectives. The consultant did not meet this group face to face or via email and the IP Manager coordinated their feedback. The reference group reviewed the inception report prepared by the consultant and their inputs were addressed in the revised inception report. It is to be noted that the questionnaires, interview guides, evaluation matrix, theory of change and sampling frame were discussed in the inception report and the consultant received feedback on all these areas. The Evaluation Management Group provided feedback on the draft report. The involvement of the Evaluation Management Group has enabled them to hone the evaluation design, to participate in the planning and analysis of the evaluation and will most likely increase ownership as well as provide credibility and strategic directions for next steps and use of findings.

2.2.2. Evaluation Work-plan and Deliverables

A work plan was developed in consultation with the IP Secretariat and milestones indicated with modifications made to the end date (as originally stated in the TOR) for the evaluation to enable realistic timelines. The work plan was part of the inception report and was approved by the Evaluation Management Group as well as the commissioner of the evaluation i.e. the SPPME unit. The planned and revised deliverables are stated below: there was a change in the final dates of submission because adequate numbers for quantitative and qualitative samples were not achieved in the time expected. An extension was required for data collection.

Date Due Deliverables Revised dates

July 2 Submission of Inception Report Same

July 18 Submission of revised Inception Report Same

Sept 20 First Draft Completed Sept. 30

Oct. 15 Final Report Oct. 15

2.2.3 Evaluation Process

To understand the IP process, interns and host organizations, the consultant attended the three day orientation for the 2012 interns. This provided an opportunity to interact with IP secretariat, incoming interns and a few past interns. Host organizations were invited for one of the days and discussions with them helped understand why they had signed up for the IP. The discussions,

18

review of available program documents, IP feedback forms, case studies and the website provided information relevant to the evaluation and guided the development of the various tools for data collection. A quantitative web-based survey using Zoomerang was also explored and as UNICEF ICO had subscribed to it, it was decided to use it for the quantitative data collection. A meeting was scheduled over the next fifteen days to meet the chief of the SPPME unit Mr. Joaquim Gonzalez-Aleman to understand his expectations of the study. A full day workshop with the IP Manager, Amrita Singh and the Research and Evaluation Specialist Michiru Sugi resulted in the development of the Theory of Change, which further enabled the honing of questions for the evaluation. Discussions via email and Skype were periodically held with the IP Manager and Research and Evaluation Specialist to refine the evaluation design, instruments and sampling. Periodic updates were also provided to the IP manager by the independent evaluator.

2.3 Stakeholder Participation

As guided by the HR principles, the evaluation involved a wide range of stakeholders in the design, planning and implementation of this evaluation. The reason to do so was to strengthen ownership and build agreement for every step of the evaluation process. This involved the active participation of the Evaluation Management Group, IP secretariat (manager and consultants), Research and Evaluation specialist as well as the Zoomerang experts within UNICEF ICO. The other stakeholders involved were the interns both national and international, host organizations, UNICEF program officers and decision makers. Interns participated in pilot Zoomerang surveys and this was found to be extremely useful to refine questions and close open-ended questions. Stakeholders were involved in discussions and consultations providing in-depth information about how the program was, its design, what worked and what did not, why and the next steps forward. The responses of the varied stakeholders enabled triangulation as well in assessing the program processes, its effectiveness and efficiency. This methodology also elicited next steps – what could be improved, how and unexpected results.

2.4 Data Collection and Analysis

2.4.1 Data: Types

As discussed earlier, the evaluation methodology included both quantitative and qualitative data to ensure strong evidence. The quantitative data included representation of interns from different countries, representation of a number of host organizations and UNICEF state offices involved, numbers of case studies published, costs involved for each of the four years under study and IP satisfaction data. The qualitative data included perceptions of stakeholders about the IP, what it achieved, the implementation process, satisfaction with the program, understanding UNICEF’s work and the use of the case studies. Data was collected from interns, host organizations, UNICEF state offices supporting the interns and other decision makers (such as UNICEF state Chiefs) regarding the implementation and outcome of the IP including reasons for what worked and what did not. Importantly, with each stakeholder group, forward-looking questions were asked regarding the continuation of the program, redesign and the engagement of the now 500 strong intern community. Both types of data were important to capture the holistic perspective of the IP, providing a rich understanding of the project’s accomplishments and lessons learned leading to a set of contextual set of recommendations.

2.4.2 Data Collection Method

19

Both primary and secondary data were collected. The secondary information was collected from documents received from the IP Secretariat. Other information was downloaded from the KCCI website and UNICEF ROSA.

The primary information included a set of interviews and questionnaires as well as observation about the IP implementation (attending the orientation meeting in Delhi for the 2012 interns). As guided by the sampling frame, a quantitative sample was drawn up for each of the stakeholder groups. In addition, telephone and face to face interviews were also conducted for key informants (KI). These included interns, host organizations – government, academic and implementing organizations, UNICEF state Program Officers, UNICEF sectoral specialist, IP secretariat’s Program Manager, UNICEF ICO Research and Evaluation Specialist, UNICEF IP Consultant (for selection of interns), head of SPPME unit, UNICEF Chief Field Officers (state level). The following table explains in detail the sample frame.

Sample Method of contact Number

Interns (50% national and 50% international) preferred option is 30% sample (questionnaires will be sent to 85 persons to address attrition)

Survey 67

Interview 4

IP Secretariat (IP Manager and Consultant) Interview 2

KCCI Management (IP Manager is included) Interview 3

UNICEF Program Officers who have supervised interns

Survey 16

Interview 3

Host organizations Survey 15

Interview 2

Government officials (request made but not available)

Interview 1

Key decision makers Interview 7

Total 109

2.4.3 Data Collection Instruments

The following ten instruments were developed to collect data:

1) Questionnaire (online survey) for Interns 2) Questionnaire (online survey) for IP Managers 3) Questionnaire (online survey) for host organizations 4) Interview Guide for Interns 5) Interview Guide for IP Managers 6) Interview Guide for host organizations 7) Interview Guide for IP Secretariat 8) Interview Guide for IP Management 9) Interview Guide for Government Key Officials 10) Interview Guide for Key Decision Makers

The interview guides and questionnaires were developed by the evaluation consultant and refined with inputs from the reference group (inception report included the tools) and the IP secretariat. The online surveys were piloted and refined based on the inputs received from the pilot group. This included language editing, inclusion and exclusion of questions and

20

development of multiple choices for open questions. The interviews were orally administered over Skype and telephone. One interview was carried out face to face.

2.4.4 Data Analysis

The mix of quantitative and qualitative data and the questioning of different stakeholder groups enabled the triangulation of data increasing its reliability and validity as well as understanding the perceptions of different stakeholder groups. In addition, cross tabulations across gender, national and international interns and year of internship (experience of older versus newer interns) provided further analysis of the data. The data analysis used simple descriptive statistics such as percentages and frequency counts to capture variation among the target groups. These then were compared across stakeholder groups across the various indicators related to each criterion. Case studies were analyzed according to criteria agreed upon during the inception phase and the ToR. Qualitative data (open ended questions) generated by the survey for interns (Zoomerang) was analyzed using a content analysis and the evaluator developed categories of responses. Data from interviews with key stakeholders were triangulated with other data. Analysis of efficiency data included a) unit cost and b) a professional analysis based on interviews. More sophisticated cost benefit analysis is beyond the scope of this evaluation.

Overall, the data was analyzed according to relevance, efficiency, effectiveness, sustainability and impact.

2.5 Challenges

The evaluation relied on the primary data collection as well as the available feedback of the orientation and the IP program routinely collected by the IP secretariat. The consultant selected the interns, program officers and the host organizations randomly for the online survey. However, there were problems in the contact details, particularly for the interns. Initially, to obtain a 68 sample of interns about 120 (out of a total of 230) interns were sent emails for the online survey. However because of the bounce back of emails, an additional 40 were sent emails. In spite of this, regular follow-up, reminders and deadline extensions were needed to obtain the 68 sample. The survey had to close at 67 persons. The same was the case for the online survey with the host organizations and the UNICEF IP managers. The IP Manager assisted the consultant in follow-up. Ultimately, it was decided to stop at one third instead of fifty per cent sampling. The TOR had not mentioned any sample size and so the consultant in discussion with the IP Secretariat made a decision to continue with the one-third samples, which was anyway more than originally expected by the IP Secretariat. The selection of stakeholders for the interviews was made in discussion with the IP Manager based on experience with the IP program and willingness for the interview. It is possible that those who are positively inclined for the IP program may have been selected. It may be mentioned that the consultant received frank and sometimes critical comments from these selected stakeholders, therefore indicating that selection criteria were not biased. Because triangulation was used, a variety of stakeholders consulted and a large random sample used the online survey, the evaluator is confident about the accuracy of the findings.

3. FINDINGS

The evaluation findings are organized according to the DAC criteria of relevance, effectiveness, efficiency, sustainability and impact of the IP. The Evaluation Matrix and the Theory of Change

21

were used to guide the evaluation. The data analysis will link these criteria to the questions raised and inform the findings. Please see Attachment four for detailed explanation of the TOC. As mentioned before, the TOC was developed in consultation with the UNICEF ICO Research and Evaluation Specialist and the IP Manager.

3.1 Relevance

The relevance of the IP is analyzed by considering

1) The extent to which the objectives of the IP program are consistent with UNICEF India’s knowledge management results (as per Country Program strategies and IR) as well as meeting the knowledge/demands of other IRs?

2) To what extent will the objectives of the program be still valid in context of new country program (IR 6.2 under PCR 4) can only be suggestive as subsequent discussions inform that the new country program is under development

3) Whether the structure and design of the program are appropriate to meet the KM results.

3.1.1 IP relevance to overall KM objectives and strategies internationally and in India a) The IP program is consistent with the KM strategies as articulated by UNICEF HQ and the

UNICEF ICO Country Program and especially regarding knowledge generation.

Knowledge management is an important strategy for UNICEF worldwide and in India in particular. UNICEF HQ concept note on KM14 lays out a broad vision expecting each country to adapt KM according to its needs. Central to the note are two points: a) KM must improve access to existing knowledge on children and adolescent rights and development; improve knowledge sharing around key focal areas; and lastly encourage innovation and new ideas to improve outcomes. The second point is b) the distinction between knowledge management and knowledge function. Whereas knowledge management is limited to the organization and delivery of knowledge, knowledge function is related to a broader set of functions which includes identifying priority areas, quality of data collection, generating and interpreting it and most important its use in both programming and policy. UNICEF India has articulated the importance of knowledge management. In the 2008-2012 Country Program (CP)15, it positions itself ‘not so much in terms of its financial resources but more in terms of sharing national and international know-how for improved program implementations’. One of the key strategies to achieve this is ‘providing technical assistance and support to improve knowledge management systems and sharing of lessons learned’.

The KCCI initiative provides expression to the KM intent and the IP program is one of the many KM activities situated in the KCCI initiative. The concept note identifies that UNICEF wishes with this positioning to be a ‘global knowledge broker of choice for expertise on children’. But it recognizes that within India there are ‘systemic weaknesses in capturing, packaging and sharing knowledge across the organization’. The concept paper also clearly states its vision that knowledge would not only be generated, but it would be shared for advocacy to influence

14

KM Concept Note draft 2 Sept. 2006 15

Knowledge Management in UNICEF India Concept Note for Discussion, January 2008; KM Concept Note draft 2 Sept. 2006

22

program and policy and that it would support strategy development for ICO itself. In this sense, the UNICEF ICO emphasis on using KM for influencing policy and program and the UNICEF HQ stress on knowledge function are compatible. Both documents indicate the importance of knowledge generation as well as use.

The IP program is unique in that it is not just an internship program but is structured in a way that produces a KM product, usually a case study (in earlier times, videos were also developed). It encourages interns to go to the field, to collect data, analyze it and present it to relevant stakeholders. There is also an emphasis on quality as all case studies are reviewed and only those that pass the test are published and disseminated to relevant government departments, libraries and other relevant stakeholders. The case studies are placed on the KCCI website clearly demonstrating knowledge generation.

3.1.2 Relevance of IP objectives with KM results

Respondents ranked the objectives as a whole as well as each objective in terms of relevance. Respondents were asked to rank each objective as very relevant, relevant and not relevant. A deeper analysis of each of the objectives of the IP program provides an understanding of how they contribute to the KM results.

b) Overall, most UNICEF managers at state level (involved with the IP) and host organizations indicated that the IP program contributed a small but important niche towards achieving India’s KM results especially in knowledge generation and to a small extent in knowledge use.

A one third sample of all UNICEF managers at state level (16 out of 40) associated with the IP program since 2008 responded to an online evaluation survey of the IP. About 81% or 13 out of 16 managers thought that the IP program was relevant or highly relevant to the KM result, which is to provide technical assistance and support to improve the KM systems and share lessons learned. They mentioned the large number of case studies available as testimony to the increase in documentation and creation of a corpus of knowledge that was not available earlier. Interviews with program managers and key decision makers indicated that KM was a large concept and the IP was only a small section of it, filling in a specific niche of KM and much more needed to be done to fill the KM space. However, the use of the case studies was not clear even though they were published and hence their contribution to KM was difficult to ascertain. Only three out of 16 or 19% of the managers’ felt that the IP program was not very relevant to the KM needs.

Host organizations who partner with UNICEF at the state level also responded to the question regarding the relevance of the IP program for the KM results of UNICEF. About 87% host organizations felt that the IP program did contribute to UNICEF’s KM results. Interviews with hosts informed that they believed knowledge generation occurred across several levels – interns learned about rural India, the project’s knowledge base increased and so did that of the host organization. They also mentioned that interns brought new data and fresh perspective, helping to document issues not normally captured in program reporting. They were not sure how UNICEF had used the case study for its programming.

These perceptions of UNICEF managers and host organizations assess the relevance of the IP as a whole towards realization of KM results. Each of the IP objectives was also assessed regarding their relevance to KM based on the perceptions of interns, host organizations and UNICEF Managers.

23

c) The IP objective of producing high quality documentation (case studies with reference to the IP) to influence government, policy makers and development practitioners was less relevant for UNICEF managers at state level, host institutions and decision makers and somewhat so for interns.

About, 31% UNICEF managers at state level and 20% (3 out or 15) host organizations felt that the objective was not relevant. About 12% (3 out of 67) interns felt the same, mentioning during their interviews that they assumed that the case studies would be used by UNICEF. Key decision makers felt that this objective for use in policy and programming was too ambitious. This first objective includes two aspects a) produce high quality documentation and b) to influence policy makers and development practitioners. It is the second part which is problematic for UNICEF managers, host organizations and key decision makers. There was agreement among all stakeholders regarding the high quality documentation (which was objectively reviewed to determine whether the case study made the ‘cut’ for publishing) but it was difficult to accept that the product produced by interns in the short time available would be used for policy making or programming decisions.

d) The IP objective of collection of field work data on selected issues to complement UNICEF’s documentation needs was very relevant for interns, somewhat relevant for the host institutions and most decision makers but of less relevance to UNICEF managers at state level.

Different stakeholders had different perspectives regarding the relevance of this objective with 67% interns rating it as very relevant followed by 53% (8 out of 15) host institutions and lowest at 31% (5 out of 16) for UNICEF managers. KI indicated that this objective was clearly relevant and that a fresh look at ongoing projects brings interesting observations. UNICEF managers indicated that UNICEF regularly documented their work either through their own human resources or hiring consultants and could do so with a much wider scope for collection of data which according to them was necessary to influence policy and programs. Qualitative data indicates that there are many options available for UNICEF to document their work and the IP was only a small part of it. Having said that, some UNICEF managers did think it was a cost efficient option to document aspects of work that would never have got documented. About 56% UNICEF managers have considered this objective as relevant (compared to 30% interns and 40% host organizations) indicating that when the data collection is well planned and the issues well identified, field data collection enables in-depth study into issues UNICEF may not necessarily document because of pressures of work. For example, understanding stigma at a personal level or documenting a project to contribute to a wider study enables the interns’ case studies to be relevant for larger issues.

e) The IP objective of developing a cadre of young professionals to understand children and development needs in India was considered to be very relevant to most interns, UNICEF managers at state level and decision makers but less so for host organizations.

About two thirds interns and UNICEF managers (64% interns, 69% (11 out of 16) UNICEF managers) indicated that developing a cadre of young professionals was a very relevant objective of the IP whereas only less than half or 47% (7 out of 15) host organizations felt the same. There is a clear mismatch of expectations from the host institutions. KI considered this objective as a highly relevant one, preceding all other objectives.

24

f) The IP objective of developing partnerships with leading research institutes and promoting further interest in development research was rated very relevant by most host organizations, somewhat by key decision makers but had low relevance for both interns and UNICEF managers at state level.

Almost three fourths (73% or 7 out of 15) host institutions felt this objective as very relevant but the same perception was not shared by either interns (39%) or UNICEF managers (31%). The host organizations look forward to the IP as a means to build partnerships with UNICEF but this is not shared by UNICEF managers who considered the linking with the host organization as limited to the IP. Some UNICEF managers mentioned that there are very few good research organizations and it is often difficult to find a host that can support interns logistically. The expectations of UNICEF and host organizations are not in synergy and the IP is considered as just one more event to be managed. Interns understandably are not interested in how UNICEF partners with the host as long as they are assisted in completing their assignment to the quality expected.

g) The IP objective of promoting awareness of UNICEF among students in India and abroad was rated very relevant by about half of UNICEF managers at state level, host organizations, interns as well as most decision makers.

It is not surprising that promoting awareness of UNICEF is important and very relevant to UNICEF managers (56% or 9 out of 16), host organizations (60% or 9 out of 15)) and interns (48%). Interns explained that they were aware of UNICEF’s work through the web but needed to get a hands-on experience in collecting data and understanding grassroots issues. One would expect higher scores from UNICEF managers considering that the internship is UNICEF driven. It appears that each stakeholder views the IP program narrowly only from their perspective.

h) There is a mismatch regarding stakeholders’ perception of relevance of different IP objectives reflecting a lack of coherence about both IP and each stakeholder’s contribution towards the program.

Except for the acceptance that the first objective of influencing policy and program decisions was too ambitious for the IP program, the other objectives appeared to be aligned with the perceptions of different stakeholders responding to their own circle of influence and work. So, a host organization considers developing partnerships with UNICEF as very relevant whereas UNICEF does not feel the same or developing a cadre of young professionals is not a vision shared with the host organization. The host organizations do not understand that one of the major purposes of the IP is sensitizing a new generation towards development and a close understanding of children’s needs in India. In fact, they see the IP as one method of building partnerships with UNICEF and not as knowledge management. Similarly, UNICEF managers do not view the collection of field data as very relevant to their documentation needs. There is a lack of a holistic understanding of the objectives of the IP overall. (Please see Attachment Five for tables and further information)

3.1.3 IP relevance to KM of other IRs

i) The IP has a strategy only for knowledge generation (production of a quality case study) and not use and for this reason, the IP is partly relevant in addressing the KM needs of other IRs.

25

The IP in 2008 worked on social exclusion across all teams but in subsequent years has worked on knowledge demands identified at state level across different sectors. The case studies are available for communication, environment, emergency, education, child protection, social policy and cross-sectoral issues (see. www.kcci.org.in). UNICEF Country Plan 2008-12 lays out IR for Child Development and Nutrition, Child Environment, Child Protection, Education, HIV, Behavior Change Communication and Advocacy and Partnership as well as Emergency Preparedness and Response and each have their own KM needs that they respond to. Interviews with KI indicate that interns often bring a fresh perspective; work that is not often documented is done so by interns, which is helpful (e.g. documentation of coordination efforts in disasters was used to influence policy and practice). Interviews with interns, UNICEF managers, host organizations and KI indicated that high quality depends not only on interns’ capacity but also on other factors such as guidance from host institution, linkage of host with data sources, and availability of data. The case studies are assessed and approved against objective quality criteria and if they pass the ‘cut’ have generated knowledge which is available for use.

But knowledge generation does not necessarily lead to knowledge use. Use and influence of a knowledge product does not happen automatically and has to be planned systematically from the planning stage. For example, the sector issue which the case study addresses will need to be disseminated as well as develop pathways for use. Interviews with UNICEF managers and host organizations regarding dissemination and use indicate wide differences – with some states and host organizations using the products to further their advocacy more than others. Some host organizations informed extensive use of case studies as teaching material for their students. However, it is not clear whether UNICEF or the interns are aware of this or what the purpose of the teaching material is or how it could be extended. For example, a UNICEF state office mentioned that the video on social exclusion has had great impact and been widely used. It also helped UNICEF to advocate to the government tactfully using an ‘external’ voice. Interviews with UNICEF managers, host organizations and key decision makers indicated that a case study (after quality approval) has been used, because of the personal commitment of UNICEF managers, host organization and key decision makers but there was no strategy for knowledge use.

3.1.4 IP relevance to new country program

j) The IP is relevant to the new country program’s emphasis on evidence based knowledge to promote issues of children’s rights but will need to develop strategies for use of the case studies.

The new country program states one of its IRs is ‘evidence based strategic knowledge and advocacy tools on prioritized issues related to children’s rights developed, documented and disseminated’ which will contribute to the Program Component Result 4: Policies, practices, programs, public opinion and social norms advance the rights of children, adolescents and women. The current IP program does collect fieldwork data on issues of priority and the new country program IR talks also of use of knowledge for strategy and advocacy and to inform program and policy. As stated earlier, a focused IP product can contribute to knowledge generation and in the new country program perhaps the element of use of the IP KM products could be emphasized.

Interviews with KI and UNICEF managers indicated that the new country plan emphasizes the need for KM and the IP will continue to be relevant with its generation of case studies and perhaps the inclusion of audio-visual and more creative methods of documentation made


26

accessible through the digital revolution and the generally high expertise to use them among young people.

3.1.5 IP structure and design relevance to KM outcome

Intended output, duration, administration, management

k) The IP design is systematically administered to produce the knowledge generation outcome of a case study but there is no explicit strategy for use.

The role of UNICEF ICO for IP program is related to the selection of interns and coordinating their placements in different states. However, the selection of the topic, quality of the host organization, quality of the case study support and its dissemination and use is with the UNICEF state office or sections that supervise interns. The IP structure and design has been designed to include external researchers (interns) to collect information (field data and literature review) on an issue of priority (TOR) supported by research organization (host organization) and technical backstopping (UNICEF state offices) to document (case study) and disseminate to key stakeholders (government, other NGOs, UNICEF) as well as increase the corpus of knowledge (through print and the web). The use of the knowledge product is based on the assumption that if the case study is available, it will be used. So in terms of knowledge generation the path for the IP is very systematic. The path is less clear for knowledge use. Both are needed to be relevant to the larger goal of KM.

The TOR are designed for a short two month project and so the intended output matches with the duration of the internship. In terms of administration, the interns are selected in a transparent manner and matched with their interests and capacities to various research areas available. The TORs are decentralized reflecting UNICEF’s own decentralized structure and hence it is the UNICEF state offices that take the lead on selection of relevant issues for research.

It is important to note that except for two years 2008 and 2009 where there was a unifying topic of social exclusion and capacity building respectively, the other years have depended on the state UNICEF office to select the topic AND the host institution. Hence, the quality for the research TOR as well as the selection of the host organization is mainly with the State Offices. This means that the onus of use also lies with the state UNICEF offices.

The TOR is presented to the interns during their orientation in Delhi and host organizations are present as well as state UNICEF representatives. Usually, this is the first time all three meet – interns, host, and UNICEF state office staff.

The teams usually undergo another orientation at state level and in varying degrees are assisted by the state UNICEF offices and host organizations for the research design, data collection and analysis. In some states, the interns get an opportunity to present to government.

The case studies are presented in Delhi at the closing workshop, comments received and case studies submitted. Two independent persons – one sector specialist and the other at state level – then grade the case study according to standardized criteria and only if BOTH agree then the case study goes for publication. This usually takes six months.

27

Although the IP products may not influence policy (and many other UNICEF documents purposefully produced by consultants probably share the same fate), there is a circle of influence that is possible. However the TORs do not mention what next and dissemination (publishing and website) are considered to be proxies for use. The intern’s products do contribute to knowledge management but not necessarily for policy and advocacy

Selection of topic and host organizations for research

l) The UNICEF state offices, sometimes with the section head, drive the research agenda by identifying the research need, developing the TOR with varying involvement of host organization and government.

In 2008, under the umbrella of social exclusion and in 2009, capacity building provided a framework for selection of research issues. In 2011, it was good practice. UNICEF ICO KI indicated that it was up to the UNICEF state office to be clear about the TOR and that the selection of the host organization should be relevant to the topic. The personal commitment of the chief of staff at state level made a lot of difference in the administration of the IP and the use of the case study.

In most cases, UNICEF state staff (with SPPME leading the process at the state level) selects the topic on perceived need and then informs the government for approval and for access to field sites. Host organizations are involved at a later stage when the research frame has been decided. Wherever this process has been methodical and the chief of field office or section has been involved, the IP program has worked well. KI stated that they got good documentation by highly talented young people with a very fresh perspective, so if the IP is used with care it does have relevance to the KM outcome. The opportunity to involve host organizations meaningfully is available with the program, but there is little dialogue to link the IP work to a larger vision of ongoing research needed by UNICEF and the host organization and to issues important to the government and the state. There was limited early consultation by UNICEF state offices with host organization and the government (also see TOC), which affected the relevance of KM. In theory, the IP structure and design is very relevant to the KM outcome to strengthen ongoing partnerships with government and host institutions but needs to be implemented in a strategic and timely manner. m) The case studies are limited to good practice documentation and do not explore other KM

needs.

The IP has been confined to documenting good practices (usually of UNICEF supported programs) but the subject of the research could be formative or inform about different aspects of a child’s life in India. Such understanding of what the IP can produce would increase the relevance of the IP program immensely. The ‘freshness’ of the interns could then be utilized more productively. This also ties in with the intention of KM as mentioned in UNICEF medium-term strategic plan, 2006-2009 Investing in children: the UNICEF contribution to poverty reduction and the Millennium Summit Agenda (Pg 12) which supports knowledge-generation and analysis on the situation of children and women to many stakeholders including but not limited to policy makers but to a broad range of government and civil partners including “those who are the voice of society, children and young people” (pg. 95)

Intern selection and composition

28

n) The program attracts high caliber interns who are selected in a transparent manner; but catchment areas for international interns are mostly North America and Europe and for national interns, Delhi and Maharashtra.

The IP program attracts interns of high caliber internationally and from India. They are shortlisted using criteria that ensure that the best interns are selected and in this way can contribute their best to the knowledge product. The interns’ selection both national and international is competitive and the following table indicates the number of interns who applied and were selected. Year No. of international

interns applicants received

No. of international interns accepted

No. of national interns applicants received

No. of national interns accepted

2008 151 45 289 38

2009 111 35 137 31

2010 175 20 200 24

2011 131 21 151 19

Internationals are predominantly from North America and Europe and in India from Delhi and Maharashtra. This is due to several reasons – past alumni are from the same region, the administrative set up in North America universities enable a one window advertisement for the internship call (unlike other universities where individual schools or departments would have to be approached). In India, aside from similar reasons as stated for internationals, interview with the UNICEF consultant who reviews applications stated that the need to have good English privileges Delhi based students since other geographical areas may not have students with similar language skills. It may also be mentioned that in selection of Indian interns the criteria is flexible since most Indian students do not have the work or research experience that their international counterparts have. The reach of the IP has been in this sense less diverse than expected. Indians carrying Indian passports but studying abroad are considered to be national or Indian students. More girls than boys apply hence the selection of the final batch reflects this ratio as well.

29

National Interns

International Interns Year Countries from where

applications received Number Male/female ratio

2008 (Total 151) 42 Countries, plus 6 applicants with dual citizenship;

49 USA and Canada 64 Europe 38 Rest of the world

26/125

2009 (Total 111) 33 Countries, plus 4 applicants with dual citizenship

40USA and Canada 44 Europe 27 Rest of the world

17/94

2010 (Total 175) 44 Countries, plus 4 applicants with dual citizenship

64 USA and Canada 56Europe 55 Rest of the world

30/145

2011 (Total 131) 34 Countries, plus 7 applicants with dual citizenship.

54 USA and Canada 40 Europe 37 Rest of the world

20/111

o) Composing teams of national and international interns and matching teams to research topics of interest is systematic and largely successful.

Teams of four are placed together composed of internationals and nationals with matching interests for the topic of study and then placed in relevant states with host organizations. This is an elaborate exercise and considerable thought is placed in doing so. A few (two or three) do drop out during the internship, some unable to manage the field experience.

p) Hosting the IP program is not compulsory and UNICEF state offices choose to sign up indicating a commitment for the IP.

The internship has been modified over the years, hoping to enable real commitment from the UNICEF state offices. From 80 interns in 2008 to 65 in 2009 to 45 in 2010 and 41 in 2011, the

Year In India Studying out of India Male/female ratio

Number Countries Number

2008 (Total 289 applicants)

Total 271: 100 Delhi 50 Maharashtra 41 Uttar Pradesh 80 Rest of India

4 18 125/164

2009 (Total applicants 137)

Total 122: 60 Delhi 10 Maharashtra 9 Uttar Pradesh 43 Rest of India

6 15 55/82


Total 183: 82 Delhi 29 Maharashtra 72 Rest of India

4 17 56/144


Total 137: 38Delhi 40 Maharashtra 59 Rest of India

3 14 47/104

30

decline is because the IP is no longer compulsory (implicitly) for state offices. This is to ensure that only those state offices who are committed will host interns. The selection, placement and state office request for interns indicates that the IP program is relevant to provide quality and need-based KM results.

Summary of Relevance

The IP program is to a great extent relevant to and able to respond to the KM needs as outlined in the various documents. The IP program contributed a small but important niche towards achieving India’s KM results especially in knowledge generation and to some extent in knowledge use. There is a mismatch regarding UNICEF and host organizations’ perception of relevance of different IP objectives reflecting a lack of coherence about IP. The IP partially contributed to the knowledge generation of other IRs; however it is difficult to track use. There is a rigorous process of selecting the best interns, nationally and internationally, use of criteria to ensure quality of the KM product (intern’s case study) and strategy for dissemination (print and web). Use is dependent on many factors of which knowledge generation is just one. The IP relevance to the new country program is clear in terms of its contribution to knowledge management on issues of importance to children. In terms of the structure and design, the IP is eminently suited for KM but can strengthen relevance through the design and management to encourage use.

3.2 Effectiveness

For this evaluation, effectiveness refers to whether the IP program achieved its original objectives and what is the evidence regarding achievement or not. To answer this question, the evaluation reviewed each of the four outputs and outcome to ascertain whether IP results were achieved. (Please see logic model diagram on Pg.16) The perceptions of all stakeholders, interns, UNICEF managers, key decision makers and host organizations were triangulated. Because the evaluation is forward looking intending to inform the next country program, stakeholders also identified how to improve effectiveness.

To study the deliverables (case study), which is an indicator of effectiveness, an in-depth study of the quality of the case studies was undertaken as well as perceptions of various stakeholders sought. Effectiveness also reviewed whether there was intention of use in the interns’ TOR or evidence of use of the case studies. (Please see Attachment Six for Tables related to this section)

Selection and expertise of host organization: Stakeholders’ perceptions

a) There are sharp differences in perceptions of interns, UNICEF managers and host organizations regarding host organization’s subject expertise, research capacity, providing feedback to interns on case studies and in communication with UNICEF and government regarding interns’ research indicating conflicting expectations.

The selection of the host institution is considered to be critical for the success of the IP program since they provide the technical input, logistic support and ensure the smooth progress of the data collection, analysis and writing of the case study. In many aspects of the host organization’s ability, there is a difference between the perception of UNICEF’s officers and the interns. This indicates a serious lack of understanding and communication regarding expectations.

31

Host organizations perceptions were very different from the interns and somewhat from UNICEF managers. The host organizations were happy with their quality in research and subject expertise, in providing feedback and in communicating with UNICEF. It is in the limited communications with government that all seemed to be in agreement although the percentage varies greatly.

Over 50% interns did not feel that the host organization was appropriate either in subject expertise, research expertise, providing feedback to the interns on the case studies or in communicating to UNICEF about the progress of the research and case study. In communicating with the government about the case study, only 22% felt that the host organization did so. These findings resonate with the feedback received from interns every year UNICEF ICO.

Was selection of the host organization appropriate?

Top number is the count of respondents selecting the option. Bottom % is percent of the total respondents selecting the option.

Interns UNICEF Managers

Host organizations

Yes Yes Yes

In subject expertise 35 12 14

52% 75% 93%

In research expertise 33 7 14

49% 44% 93%

In providing feedback on case studies 32 14 13

48% 88% 87%

In communication with UNICEF about the progress of the research and the case study

28 15 13

42% 94% 87%

In communication with government about the progress of the research and the case study

15 7 9

22% 44% 60%

The table above clearly indicates a serious difference in quality expectations and a lack of communication between all three stakeholders.

b) All stakeholders, UNICEF Managers, host organizations, key decision makers and interns, have similar suggestions to streamline host-UNICEF-intern relationship regarding management of interns’ study, clear and consistent communication about study objectives amongst all stakeholders, use of knowledge product and providing the space to interns to independently express their findings.

To strengthen the host organization, the interns have clearly indicated what role the host organization must plan and unanimously have indicated UNICEF’s leading role in explaining the objectives of the study to other stakeholders. Intern interviews state the urgent need for quality research institutes and a mentor who is dedicated, interested and competent for the internship. How can host organization’s contribution to the Internship Program be strengthened?



Host organizations

Yes Yes Yes

32

Have ONE mentor in host organization who is a point person, easily available and able to give guidance and take suggestions from the interns

63 16 14

94% 100% 93%

Ensure goals and methods of host organization align with UNICEF

58 10 13

87% 62% 87%

Ensure clear communication between host organization and UNICEF and other key stakeholders about objectives of study

67 15 12

100% 94% 92%

Host organization to clearly state Dos and Don’ts in first week of internship – for conducting the research, relationship with field NGO, personal issues of interns and expenses related to project study and travel

60 16 14

90% 100% 93%

Host organization should help interns to liaise better with government

58 13 11

87% 87% 73%

Host organization should understand their role is ONLY of a mentor and must allow the interns to write the findings independently

58 14 12

88% 88% 80%

Host organization to involve interns in meetings with NGOs, academia, seminars where applicable

63 16 14

95% 100% 93%

Independence in reporting findings was important for all – interns, UNICEF and the host as indicated in ‘Host organization should understand their role is ONLY of a mentor and must allow the interns to write the findings independently’. However, qualitative interviews indicated difference in interpretations of ‘independent reporting and analysis’. Whereas UNICEF and the host have concerns of conveying difficult information to government authorities, interns value the need to express data transparently. Interns have mentioned that host organizations do not share data, that in some cases national interns have been reluctant to portray religious cleavages, and in other cases host and/or UNICEF have tried to influence the data analysis. According to interviews of UNICEF managers and host organizations, interns do not fully appreciate the context and constraints under which the programs operate and it is not the sensitive data but how it is represented that is important and for which interns may not have the right judgment.

Interns’ commitment and expertise for quality knowledge product:

c) Limited participation of government and host organization in developing the TOR was seen as a gap especially as almost all interns and UNICEF and most host organizations felt that TOR development process and quality were key for a quality case study. The quality of the research (case study) is dependent on a high quality TOR that is developed through dialogue by key stakeholders. In development of the TOR, the Managers indicated that the following were involved: UNICEF State Office (89%), UNICEF ICO (79%), host organization (60%) and government (25%). The host organization response to who was involved in the development of the TOR indicated the following: UNICEF ICO (100%), UNICEF state office (86%), host organization (60%) and government (54%). Clearly, the host organization understood the inputs were driven by UNICEF and especially ICO and that in half the cases the government was involved. However, in both cases with varying percentages, there is a common perception that the TORs are UNICEF driven with some involvement from host organizations and less from government. It is interesting to note that host organizations assume that government is involved half the time

33

whereas UNICEF managers mention government is involved in TOR development only one fourth of the time.

It is important to note that the KM products are meant to inform policy and practice, but the TOR development minimally involved the government and to some extent the host organization

Strengthening research design and sampling was important for 97% interns, 100% UNICEF managers and 80% host organizations. The research topic must be very specific regarding the scope of work and expectations from interns and that it should involve on the ground field research and not merely secondary data collection.

All interns indicated that the host organization must have the capacity to mentor interns and that the research topic must be very specific in scope of work and expectations from the interns (97%). UNICEF managers mentioned that though the robustness of the research design is important, the capacity of the host organization to mentor the interns is very critical.

d) Gaps in research skills of interns and host organization were identified as important constraints for a good case study with Interns requesting additional need based research skills training and host organizations suggesting additional personnel to support interns with research and statistical skills.

Interviews with UNICEF managers, host organizations and key informants indicated that interns had theoretical skills but application to real settings required mentoring. There was a difference between international and national interns, as international interns in the team were likely to have more research background. About 82% interns have suggested including optional additional workshops on research skills such as GIS mapping, interviewing techniques, data visualization and analysis whereas host organizations (93% or 14 out of 15) and about 75% of interns and UNICEF managers have suggested the need for a research person or statistician available for the interns. Interviews with managers indicated the difficulty of finding research institutions that were of high quality and also had field experience or finding implementing organizations with research capacities.

e) UNICEF’s regular follow up during data collection and feedback on interns’ draft case study was considered more important by UNICEF and interns than host organizations for ensuring a quality case study.

Over 90% UNICEF managers and interns compared to 67% host organizations (10 out of 15) mentioned that regular follow-up by UNICEF ensured quality of the case study. Although 73% (11 out of 15) host organizations perceived UNICEF’s feedback was very important to improve the quality of the case study, both the interns (86%) and UNICEF (94% or 15 out of 16) were clear that UNICEF involvement as a technical backstop and quality control was important. Most interns (81%), host organizations (73% or 11 out of 15) and UNICEF (94% or 15 out of 16) also felt that host organizations should provide feedback as well.

f) Interns and UNICEF managers were less satisfied with the quality of the case study than the host organizations.

The interns were divided regarding their satisfaction with the case study with only half (45%) reporting satisfaction. Only 30% of managers were very satisfied with the case studies (with 56% somewhat satisfied) expecting much more from the interns. The host organization was generous, 73% saying that they were happy with the quality of the case study.

34

g) UNICEF managers and interns identified gaps in criteria assessing quality of the case study although UNICEF Managers, interns and host organizations mostly agreed on the adequacy of the current criteria to assess quality.

The following criteria were used to assess the case study: adherence to TOR, good understanding of broader context, describes value added to current programming practice, process and results describe actual changes in physical condition/behavior/attitude of project stakeholders, quality and source of quantitative and qualitative evidence, strategy and implementation describes successes and potential for replication. Except for adherence to TOR (interns mentioned this only 77% of the times, since they felt the TOR were themselves often flawed), all other criteria were considered very useful scoring over 90%. Host organizations and UNICEF managers also felt the criteria were appropriate except that UNICEF viewed adherence to TOR more highly (94%).

In terms of satisfaction with the criteria 80% of host organizations were satisfied but only 60% UNICEF Managers, and 50% interns were happy. Some of the suggestions by interns for additional quality criteria were in order of importance – robustness of research methodology, use of existing literature on the topic, original evidence and information available and replication of the research design. The UNICEF managers laid greater emphasis on original evidence and information available rather than the research design whereas only half of host organizations emphasized robustness of research methodology.

Once again, there is a difference in the expectations of the interns and the host organizations primarily. Interviews with the interns indicated that what attracted them to the UNICEF internship was the possibility of doing research on the ground and they were keen on exercising their skills. Clearly, the host organization, perhaps because they are less skilled in research or viewed the IP differently in terms of the emphasis on research, were more interested in collection of original data, use of existing literature and replication of the research design. One intern mentioned how they arrived to collect data on transportation for tribal children to facilitate access to schools only to find that within fifteen days of their arrival the schools had closed, questioning their research methodology!

The evaluator did a qualitative analysis of a sample of four case studies from 2008-12 and suggests that the following would improve the case studies: a) inclusion of TOR b) better executive summaries c) providing an upper limit to the number of pages accepted d) a discussion on why the topic is relevant e) actionable recommendations targeted to each stakeholder and f) a section on how the findings could be used.

h) Dissemination and advocacy for use of case study has not targeted key decision makers, the government or community.

In order for case study findings to impact government program and policy, it is important that the case studies are disseminated to those who are likely to use it. This would mean the government since UNICEF usually works closely with government programs and interns’ projects usually study them. However, for the case study dissemination, 60% government officials, 84% UNICEF state officials and 82% host organizations were present. It is surprising that the numbers for both UNICEF and host are not closer to 100% since the interns are so closely interacting with both the UNCEF state offices and the host organization. There was no evidence of dissemination at the field or grassroots.

35

On the occasions when the case study was disseminated, interns, UNICEF managers and host organizations indicated that overall the interaction was brief and so the reactions to the findings were not substantive. Both UNICEF managers and host organizations conceded that their own interaction during dissemination was very brief and so was that by government.

Interns and UNICEF managers felt that UNICEF (about 50%) was more likely to use the findings than either the government (37% according to interns and 25% according to UNICEF managers) or the host organization (11% according to interns and 25% according to UNICEF managers).

Host organizations felt that government did not critically assess or appreciate findings and would not utilize the findings at all. Interns and UNICEF managers felt that UNICEF had appreciated the findings (58% according to interns but only 33% according to UNICEF managers) and so had the government (about 30%).

However, projects were scaled up, continued or stopped before the case study findings were available by UNICEF, host organization and government at least a third of the time. This indicates that findings did not necessarily lead to decision making.

Interesting to note is that only 50% managers promised to utilize findings and only a third appreciated the findings and believed the host organization and government felt the same. The managers themselves felt that UNICEF’s involvement during the dissemination was very brief, more than the hosts or government!

Whether it is the interns, host organizations or UNICEF state offices it is evident that in all areas whether in promising to use findings, or providing adequate time for the debrief or to make decisions based on findings or even to critically assess and/or appreciate – there has been limited involvement of government and in many cases UNICEF and the host organization as well. Also, if the decision to scale up the project or not was already made, one would question why TOR of such topics were selected? What was the purpose of the case study?

Summary of Effectiveness

There were limitations in the ability of the host institutions to support the interns in the knowledge and skills needed for the development of the case study. Host organizations need to be selected according to their research capacity, content knowledge and ability to have a committed mentor for the interns. The host should facilitate and guide and provide the space for interns to complete their research. The quality of the case study was dependent on the clarity of the TOR, strength of the research design and timely support of both UNICEF state offices as well as the host. Limited participation of government and host organization in developing the TOR was seen as a gap. UNICEF’s regular follow up during data collection and feedback on interns’ draft case study was considered important to ensure quality and additional quality criteria were identified. Streamlining host-UNICEF-intern relationship regarding management of interns’ study, clear and consistent communication about study objectives amongst all stakeholders, use of knowledge product and providing the space to interns to independently express their findings were important for the effectiveness of the IP. Dissemination and advocacy for use of case study has not targeted key decision makers, the government or community.

36

3.3 Efficiency

The efficiency question was addressed in the following ways:

To what extent was the IP cost effective? Is relationship between program costs and program outputs reasonable? Were measures taken for efficient use of resources? To what extent was there resource contribution from other stakeholders?

3.3.1 Cost effectiveness of the program in relation to program costs.

a) The following figures provide program costs for the four years and may be divided by the number of interns who participated. The management cost includes 70% of the IP manager’s time, 10% of the Research and Evaluation Specialist and 5% of the head of SPPME time. A consultant is hired for selection of interns to ensure objectivity and another to assist the IP Manager during the two months of the interns stay in India. It is estimated that for two months, the UNICEF state officer (SPPME) is likely to spend 10% of the time. Considering this a constant, the figures below indicate that over the last two years the cost per person has remained stable in spite of rising inflation and costs. The cost per person is indicative as there is no counterfactual and was calculated to determine the change in costs, if any, over the four years of the evaluation.

Program cost (USD) excluding personnel cost UNICEF ICO and state office

217,300 278,799 154,516 142,069

Year 2008 2009 2010 2011

No. of interns 80 65 45 41

Per intern cost 2,716 4,289 3,433 3,465

Other measures taken to increase efficiency were to reduce the number of case studies printed and offering a digitized version for dissemination.

b) Stakeholders such as interns and host organizations contribute to the IP resources. The interns particularly the international interns fund their own travel to India and the rising costs of international travel have not affected the program. The stipend for two months is modest and has not increased. Also, the host organizations have been receiving the same amount of funds that are quite modest to house the interns and for related expenses of fieldwork. Each organization get USD 6100 and no incentives are provided to the mentor.

3.3.2 Cost effectiveness of the program in relation to knowledge generation output IP program outputs relate to development of knowledge products that are used for documentation, program improvement and policy advocacy. Interviews with UNICEF program managers and questions related to whether the IP program should be continued, indicated that although involving interns had limitations of time, specialist expertise and contextual understanding – the case studies continued to be a cost effective way to research and document. Interns brought a fresh perspective to issues. Although UNICEF could hire consultants for the same job, it was likely to be more expensive and time consuming since contractual arrangements need due diligence. The short time i.e. 10 weeks of the IP placed pressure on the development of the case study and the whole process from beginning to understand the context to data collection to writing the report was a huge rush and host organizations, UNICEF and interns

37

questioned the short time available for the completion of the case study which had to pass quality standards. These comparisons related to cost effectiveness are based on perceptions but important for the evaluation. Interviews with UNICEF Managers and key decision makers indicated that interns’ use of digital technology in documenting was useful.

Summary of efficiency

The IP program may be considered as efficient as the cost of implementing the IP program has remained more or less consistent over the years. Stakeholders such as interns and host organizations contribute to the IP resources. The interns particularly the international interns fund their own travel to India. The host organizations receive modest support to house the interns and for related expenses of fieldwork. Case studies, a KM product, are cost effective considering the time and skills available to produce them by a team of young professionals using original field data. The short ten week program puts pressure on all stakeholders to complete the case study any way they can.

3.4 Sustainability

The question asked was if MWCD or other relevant institutions could fund the program. Since no person from the MWCD was interviewed and even if they were it would be difficult to answer this question and is outside the purview of this evaluation. This was mentioned in the inception report. Would other large donors be interested? Or could UNICEF HQ include the IP in its global internship program? This could be explored but this too was beyond the scope of the evaluation.

In terms of managing the program, there could be discussion on where to house it so that it becomes institutionalized. The KI informed that in general, internships in India are gaining acceptance and the concept is no longer new. However, the manner in which this IP is structured has a number of unique points that would need to be replicated appropriately such as objective selection of interns, matching their research interests with states and host organizations and the management of a KM product. Each of these has quality milestones which are important if one is to provide as smooth an experience as possible.

Having said that, the process is also streamlined and systems for contracting, orientation, selection and so on are in place. The benefit of searching for a new ‘home’ would be to institutionalize it in India and UNICEF to provide oversight. As seen earlier in the discussion on effectiveness, there are gaps in implementation in that the states need to respond earlier and with greater clarity and commitment to the IP process and in involving the ‘right’ host institutions. Any other ‘home’ dedicated to the vision of the program and quality assurance could manage the program equally well and ensure ownership by UNICEF, host organization and government.

The IP is clearly relevant and unique in its design and intent, and no other UN or UNICEF country office has a similar IP program. Sustainability however demands that UNICEF ICO promote the IP – not just for KM but in developing a cadre that are ambassadors for UNICEF and especially India. Interview with UNICEF key decision makers informed that the IP program in its current design is unique to UNICEF India and that UNICEF ICO should strengthen the IP ‘brand’ design. From this perspective, the sustainability question should be how UNICEF can continue to demonstrate their commitment to develop young professionals in a systematic way that generates knowledge as well as use. Currently, the IP is part of the KCCI with emphasis on

38

production of case studies with little emphasis on promoting use and yet intends to influence policy and practice. It is not paying adequate attention to either promoting use or to the nurturing and capitalizing on the talent that it has built over the years, over 400 such interns since 2005. There is a body of interns scattered over the world that have not been tapped and could respond to KM needs if required.

Summary of Sustainability

The IP program is sustainable since its systems are in place and has involved a variety of stakeholders at different levels, country, state and local. The IP program of UNICEF ICO has a unique design of cross-cultural teams working at grassroots to produce a knowledge product – however, there is little evidence regarding how the case studies are used. UNICEF ICO can continue to run the program. The evaluation could not explore if the program could be ’adopted’ by others since these decisions are dependent on many other factors. Currently, some parts of the IP are outsourced but fully outsourcing selection of interns, teams, matching with organizations, and reviewing case studies requires dedicated staff and expertise wherever it is housed. The evaluator can only opine that maintaining quality standards is important and that the outsourcing must weigh efficiency considerations with quality maintenance. Whether government can take it over is a moot point since the many pieces that make the IP program work are unique and difficult to replicate to the quality and flexibility needed if in a bureaucratic environment.

3.5. Possible Impact

The impact question addressed: a) the impact on interns, whether they have continued working on issues related to children or development in general, b) the use of the case studies and c) knowledge networks created with research institutions and universities d) UNICEF ICO network with interns and d) whether the overall goal of the KM was reached. Was the IP the most appropriate way forward especially as UNICEF is being requested to do more upstream policy work? This impact question was answered by considering to what extent the IP objective contributed to the knowledge management results and was done so by asking if the IP should be continued or not. In other words, what possible impact has the IP program created that would justify UNICEF involvement or not. As mentioned in the inception report, the impact question is not based on a counterfactual and follows the questions raised in the TOR. (Please see Attachment Seven for more information and tables)

3.5.1 Impact on interns:

a) Most interns stated that the IP had increased their opportunities for development work. Whether UNICEF’s IP program had solely contributed to future plans of interns is difficult to ascertain but an overwhelming number (87%) interns stated that they were better prepared to enter the development field, had a better understanding of development needs in India (96%) and understood UNICEF’s work (82%) better. For over 93% interns, the IP had strengthened their CV, and over 70% stated that UNICEF’s IP had opened doors for academic and work opportunities, encouraged them to work in India and enabled them to learn skills regarding how to design and implement a quality research study.

Qualitative data through the interviews and open ended questions regarding the benefits of the KM program consistently indicate that interns have considered the IP to contribute to their own

39

knowledge building and commitment and the production of a knowledge management product that can be shared widely.

b) Unexpectedly, the multi-cultural teams brought greater understanding of different perspectives as did the hands on grassroots experience.

The interns indicated that the mixed team of nationals and internationals contributed immensely to understanding multi-cultural and multi-disciplinary viewpoints, that the field work helped to engage in the practical use of research and understand the development challenges in India and that they were able to produce a concrete product, the case study, bringing their own fresh analysis to ongoing development problems. Interns stated that it took time for the teams to understand each other but as time progressed, they were able to capitalize on each other’s strengths. Interns have stated that the IP ‘was a life changing experience for them’.

c) Except for the selected case study posting on the KCCI website and distribution to state offices, host, interns and some interested parties, dissemination and use was dependent on personal interest and commitment.

Interns tend to share their work with friends, other interns (72%) and for employment (47%). There is some use for Indian government officials (as part of their dissemination at state level) (33%) but not used much in seminars (20%) or their own academic institutions (40%) to promote knowledge of their experience in India and only partially (34%) for their own referencing and dissertations. IP Managers report more extensive dissemination which is promising, but they too are less likely to use the case study for seminars (67%) and more knowledge sharing with educational institutions (78%). Dissemination is stronger with civil society (92%) than with the government (77%). However, the jump to use is less likely unless it is targeted. In the development of this questionnaire, it was evident that dissemination was considered equivalent to use. Hence, when asked whether the findings of the case study were used for program design or policy, the evidence is less forthcoming. Some UNICEF state offices have used some case studies extensively such as social exclusion but this appears to be the personal commitment of the chief of staff at state level or other sector heads (such as for emergency and disaster) to engage with relevant decision makers in government or civil society. The IP Manager reported that in 2008 a Rajasthan case study comparing impact of a program in two districts was used by the state and that Karnataka has used case study findings to strategize on scale up but such use is scattered and there is no concerted strategy for use.

Host organizations disseminate to educational institutions (including their own) (54% or 7 out of 15) and other NGOs (57% or 8 out of 15). Interestingly, dissemination is limited with government (38% or 5 out of 15) and in seminars (40%). Clearly, dissemination at host organization level is less than with UNICEF.

There is no strategy for dissemination and use in the TOR. The interns’ products are generated and disseminated but use is ad hoc and not tracked and therefore difficult to estimate how it contributes to the KM objectives, particularly knowledge function. Further use is also not known with 70% interns mentioning that they do not know if any other use has been made of the case studies. A feedback loop does not exist in UNICEF to indicate when case studies were used. Of the 9 interns who reported use, the case studies were used by UNICEF state office, or as reference for their own work and for their dissertations

40

Impact regarding strengthening communication and knowledge networks with interns d) There has been modest communication between UNICEF, host organization and interns

after the completion of the IP.

Professionally, 62% interns have continued their communication with other interns, 39% with UNICEF and only 18% with the host organization. It is important to note that over 94% interns would like a deeper engagement with UNICEF and have given concrete suggestions. Although a Facebook page exists, UNICEF has not actively used it with a well designed strategy to mobilize the 400 plus intern alumni.

UNICEF managers also indicated that only half were in communication with past interns but 70% did feel it was useful. Only 20% host organizations are in contact with interns but an overwhelming 93% felt it would be useful.

Interns, UNICEF managers and host organizations suggested several ways of communication The suggestions indicate that UNICEF has to take a leading role to ensure that intern networks are vibrant.

Impact on creating knowledge networks with research institutions

e) Certain institutions have been long term partners with UNICEF’s IP and there is evidence for further partnerships but there is little evidence of the IP program contributing to increased knowledge networks.

Some institutions such as Yashada in Maharashtra and new partners such as NDMA (National Disaster Management Authority) have a plan for interns and in the use of case studies. However, there was little evidence in the TOR or information received from interviews that there was a plan to develop knowledge networks with research institutions. One of the problems is the availability of quality research institutes in many states and often NGOs, who are engaged in implementation and/or action research, host interns. UNICEF managers and key informants have mentioned that concerted efforts are needed to interest quality research institutes to be involved in the IP and for other long term work.

Impact on knowledge building: UNICEF Managers and host organization’s perspective

f) The IP contributed to UNICEF’s, host organization’s and case study knowledge building but less so for government.

UNICEF managers’ and host organization’s perception is that IP contributed to UNICEF’s (about 80%)and case study subject knowledge building (94% according to UNICEF managers and 80% according to host organizations). Both were skeptical that the case study would assist in government’s knowledge building (about 40%) or strengthening collaboration with them (about 25%). According to UNICEF managers and key decision makers, UNICEF has a long term relationship with government and the IP was only one small activity in relationship building. For this reason, it would be inappropriate to say that the IP played a major role in the government’s knowledge building or to strengthen collaboration with them. Host organizations felt that they benefited in their own knowledge building (87%) and in strengthening collaborations with UNICEF (93%).

Continuation of the IP program

41

g) There is overwhelming support towards continuation for the program by all stakeholders.

94% of UNICEF Mangers and host organizations and 99% of the interns believed the IP program should continue and not be closed down.

Interns’ comments included

I personally had an amazing experience, in terms of academic learning and also as an incredible life experience. Perhaps some interns were not so ready to live in challenging circumstances or work as a team.

I would rate this as one of the best programs. It should be continued and not stopped. It gave me wonderful opportunity to work with UNICEF. There may be problems but can be easily dealt with.

My team had students from diverse academic backgrounds, which seemed detrimental at first but in the long run it was beneficial. We all brought different strengths, and we looked at the problems differently, which benefited our work.

UNICEF managers’ comments were

It is a win-win situation. We help young people get their hands dirty – expose to field realities from academia and see how the hypotheses they learned in university gets translated. They learn about the concerns of women and children and we generate their interest.

It is good for UNICEF to interact with young people and UNICEF should never lose sight of that as that is the future.

There were many comments why the IP should be continued.

Should the IP be continued?



Host organizations

Yes Yes Yes

Overall excellent program that is professionally managed 50 10 12

81% 71% 92%

Internship program provides decent stipend which enables diverse students to participate

46 7 10

73% 54% 77%

Program needs to be expanded and not shut down

58 12 11

92% 86% 85%

Youngsters understand grassroots, are able to interact with the experts and get exposed to development issues

55 12 12

89% 80% 92%

The internship program was life changing and altered life goals to want to do something for the society

34 6 9

55% 46% 82%

Internship gave a good experience of team work 58 12 12

92% 86% 92%

42

Internship gave experience on how to work with bureaucrats

45 7 8

73% 50% 62%

Unexpected outcomes were the multi-cultural team experience and learning how to work with bureaucrats. UNICEF managers mentioned ways to improve the program stating that to obtain quality, there has to be close supervision from state offices. KI mentioned that we are ‘creating ambassadors for UNICEF’ and that this is an opportunity to ‘advocate and change the minds of young people’ and at ‘one stroke, you can influence young minds from different parts of the world to know what are the different problems’. KI mentioned that although human resource intensive, in the long term, the dividends are rich.

Interns provided concrete recommendations for improvement related to host institution selection and intern management.

Interns’ qualitative responses included suggestions for selecting competent host organizations, clarifying their role as advisor, to enable the interns to report truthfully, to have closer communication with UNICEF state offices to ensure quality of case study and to have several social media ways of sharing knowledge besides the case studies. Some comments by interns are:

The host organization must be able to provide technical support and mentor students during the program. We lacked adequate mentors who had the skills and times.

Host organizations should not dictate how the case study should be carried out but to act as a resource/advisor for the interns as they design the project.

The TOR needs to be clearly written…. And expectations clearly defined.

The research project given to our team was on a training component which had taken place a year ago. So the feedback was superficial and vague. UNICEF must think of the research areas more efficiently. We did learn how to work with government officials.

Communication with the UNICEF state offices should be stronger - regular bi-weekly brief meetings with UNICEF state office would provide information and solution to issues.

UNICEF needs to have an in-house research support team that can guide the interns on sampling and research methodology, as host organizations may not have the expertise.

Besides publishing and even for those case studies not published, alternative knowledge sharing would be useful such as blogs and intern diaries.

93% of the host organization, 73% of the interns and 53% of the UNICEF managers wanted to continue the program AS IT IS without change.

Summary of Possible Impact

Overall, the IP program has contributed to interns’ awareness of development issues with field experience, ability to develop a knowledge product such as the case study using primary data

43

collection and unexpectedly about how to work in multi-cultural and multi-disciplinary teams. The case studies were disseminated but lacked an explicit strategy for use. Expecting that interns’ case studies can influence policy is unrealistic even when the KM products are vetted for quality. Overwhelmingly, the interns, host organizations and UNICEF have suggested that the benefits of the IP justify its continuation.

4. OVERALL CONCLUSIONS, LESSONS LEARNED AND RECOMMENDATIONS

Conclusions

The IP program is relevant with knowledge management strategies and has by and large fulfilled the objectives of the IP program. The structure, design and systems regarding selection and placement of interns are in place. Due to the effective management and structured design of the IP regarding linking a KM result with developing a cadre of young professionals, there has been a continuous demand for internship over the years even though the numbers accepted have decreased. As said earlier, the UNICEF program is indeed unique in its structure, design, and deliverables. Also, an increasing number of products – case studies and videos (in earlier years) are being accepted for publishing (because of passing the quality criteria). The transparency and the streamlined selection process have ensured that the best interns are selected. Dropouts have been few. The design is efficient with contributions from all stakeholders, interns, host organizations and UNICEF. Costs have not increased substantially over the four years of this evaluation.

The IP program promoted multi-cultural and multi-disciplinary understanding and resonated with the international mandate of UNICEF. The diversity in academic backgrounds and levels of expertise of national and international interns proved to be beneficial although initially interns and UNICEF found it difficult to manage. The learning curve for adjusting to a new environment and culture considered a drawback initially by state offices and host organizations streamlined during the internship because of the direct engagement of the interns with field realities.

Systematic, transparent and substantive communication has emerged as a key factor for the successful implementation of the IP. This includes the importance for regular communication a) between UNICEF state offices, government and host organization not just about logistics but of more substantive issues related to the TOR and case study b) between UNICEF state offices, host organizations and government to coordinate and select the most appropriate areas of study considering the two month period available c) between UNICEF ICO and the state offices regarding quality of research and guidelines for production of a quality product by interns and not just about contracting host organizations or managing logistics d) between interns and UNICEF regarding enabling transparent reporting of findings and how they can be represented in the case study e) between UNICEF ICO, state offices, sector specialists and leadership for the more substantive issues of the IP such as how to use the product (also depends on what is selected for research and its quality) and finally f) communication with interns both in the short term before their arrival ( and connecting with other interns past and present, proposed state offices and host organizations), immediately after the internship to communicate the status of publication (or not), information regarding use of the case study and in the long term after they move on in their careers, to catalyze on their India experience. Interns’ keen desire to continue to contribute to the IP program needs to be actively and strategically exploited – having a Facebook page is only one step in the process.

44

UNICEF ICO, state offices, host organizations, government and interns’ collaboration and shared expectations affected the case study quality and use. If any of these five stakeholders have different expectations, then it is impossible for the interns to produce a case study fit for publication. . Managing expectations was important including the host and UNICEF state offices ability to provide the interns the ‘space’ for honest data collection and reporting (and that one can learn from what does not work well) and interns for representing data in the context in which it is collected. Enabling the government to listen to and be informed by young dynamic interns is also a learning curve for both and can be tactfully and strategically used to share information that may be difficult for UNICEF and/or host organizations to report to the government.

A network of research organizations has been created over the years and it is important to nurture this relationship as well as to continually add new organizations to the pool. The IP lesson is that in short term relationships where the benefit is perceived in only one direction (for the interns and UNICEF), engagement is likely to be limited and be compromised in quality. To liaise with the best research organizations available – academic or otherwise – requires a long term relationship perceived to be of mutual benefit, contributing to the host organization’s research agenda and selecting a mentor who is motivated and enjoys working with interns. Interns have demanded better quality of mentors and research organizations with more involvement of UNICEF to guide the research process.

Impact of the IP on Interns, host organizations and UNICEF and recognition of the benefits is substantiated by the overwhelming recommendation for the program to continue. The IP has enriched interns in many ways – being better prepared to enter the development field, understanding children and development needs in India, able to network and make useful contacts and being able to apply their research skills. As one intern mentioned ‘one of the great things about the program is the creative freedom within a framework of scientific rigor’. UNICEF recognizes the role in its own knowledge building and although the request for hosting interns is now voluntary, the demand has remained constant from the UNICEF state offices indicating they do value the IP. Host organizations have added to their own knowledge building through the IP program and believe that their relationship with UNICEF is stronger. Although government is consulted and information shared, their engagement is limited for various reasons.

It may be said that the IP can be considered as an appropriate approach for KM that has contributed to KM results though unlikely at the policy level. Many reports by experts also do not influence policy. If well planned the IP case studies can lead to knowledge building in a variety of ways for UNICEF programming, host organization knowledge development and interns’ use in their own work. Government officials would be able to use the findings if presented in a friendly format and if the study area is considered as contributing to a felt need for more information. However, dissemination is not use and posting case studies on the KCCI website or distributing hard copies to relevant stakeholders does not ensure use.

Sustainability of the IP program whether by UNICEF or others will depend on many factors including how UNICEF positions the IP as a program of value for knowledge management. It will depend, besides recommendations for improving the program, upon how much UNICEF leaders, staff at country and state level believe in the interns’ ability to conduct research, host organization’s to mentor them, government to support the IP and the overall quality of the case studies that encourages not only dissemination but a clear strategy for use.

45

The evaluation draws attention to the critical role of the UNICEF decision makers, SPPME, IP Manager and the Research and Evaluation specialist to be more involved with state offices and sector specialists in the substantive and quality design of the research process; involving and energizing the UNICEF state offices; reaching out and engaging the best host institutions and raising the profile of the IP program.

The IP contributed knowledge generation but not necessarily use, was able to engage young people in learning and understanding the realities of development in India, to bring new perspectives and unexpectedly built knowledge and partnerships across nations.

Lessons Learned and Recommendations

Lessons learned and recommendations for the design of the new IP, based on the findings and conclusions, are described below.

The recommendations are clustered around three broad categories: strategic, programming and operational issues. The responsibilities for their implementation have also been mentioned.

Strategic Issues

Lessons learned with respect to relevance were 1) The IP program is relevant to and responds to the KM needs as outlined in the various documents. 2) Although there was knowledge generation, and knowledge products disseminated, knowledge use was limited including with government, interested partners in similar issues and UNICEF. Dissemination is not use and strategies are needed to create pathways for the creative use of the publications by all stakeholders including interns themselves and the academic institutions they come from. A product by interns over a two month period has to have realistic expectations; it is unlikely to influence policy (as expected in the IP objective). 3) There is a lack of coherence regarding the different objectives of the IP by UNICEF and the host organizations who mentor the interns. Some of the objectives of the IP and their order of importance need to be reviewed. 4) Interns, host organizations and UNICEF staff do see value in the design of the India IP program but it has not been shared widely. The multi-cultural team, partnerships with institutions, field work and production of a case study are unique features of the IP. The production of either print or more visual knowledge products that is vetted for quality have been appreciated but not used and disseminated widely.

Recommendation One: All stakeholders to strategically disseminate case studies and decision makers, especially in UNICEF, to advocate for use

Use is dependent on producing a quality product. The interns demand for a high quality research experience resonates with KM objectives of UNICEF ICO. Most of their suggestions to strengthen the IP program are related to developing a rigorous TOR, high quality of research interaction with the host organization, technical backstopping by UNICEF and the desire that their products – the case studies – be used. The strong demand from interns is available and provides a platform for UNICEF to capitalize on the interest of the interns by strengthening the IP process –collaborative generation of the TOR, closer engagement of government, appropriate selection of host institutions, backstopping for quality by UNICEF and promoting the dissemination and use of the case study. This would require considerable revamping of how the IP is managed and strategies developed for the selection of need based research area and the host institution; strategic engagement of the ICO and state UNICEF offices; and collaboration with the government.

46

Use does not happen automatically and will require both strategy and advocacy. A strategy for use and advocacy by interns, UNICEF, host and government has to be defined right from the beginning when the research area is defined. Ensuring that use happens then becomes the responsibility of all mentioned. UNICEF decision makers must take responsibility for ensuring advocacy – who advocates to whom and when is as critical as the development of a quality knowledge product. In communicating with government for program improvement and policy, UNICEF will need to take the lead role. Communicating to all stakeholders when use actually happens is critical. Currently the KM product is confined to print. In 2005, videos were also developed but were discontinued because of the high cost of equipment. In 2012, digital technology has changed and interactive technology widely accepted encouraging the use of multi-media for reporting.

Suggestions to strengthen dissemination and use are fairly easy to implement. Some are software related including improving the KCCI website and others are in terms of targeting the right users in the right format. For example, developing fact sheets for government and using the information for UNICEF internal planning.

Suggestions to strategically disseminate and increase use are: Search engine optimization so that academic research searches on similar topic leads to KCCI case studies on the first Google search page which only needs tweaking such as PDF files name is same as case study title; disseminate through online magazines and social media such as Facebook and Twitter so IP reaches out to a large audience and contributes towards development discourse; include a strategy for dissemination in TOR for interns and other stakeholders; provide platform for interns to present their findings on return to parent Indian or international institutions; formally connect these Indian and International institutions; improve KCCI website so that all case studies are available, they open and ensure that they are quickly uploaded for dissemination; UNICEF to inform interns through newsletter or other means how case studies have been used; disseminate executive summary to key stakeholders such as government and have a presentation of case study at state level for relevant government officials who are likely to use findings.

Recommendation Two: UNICEF to revise and reorder objectives of IP to realistically reflect what it can achieve and to manage multiple expectations

UNICEF needs to be clear about the goal of the internship. Whether interns can produce a quality product that can influence policy needs a realistic review and the objective modified accordingly. It would be useful to order the objectives in terms of importance and to share these widely with relevant stakeholders so as to manage expectations. For example, is the IP only to produce a report of publishable quality or is it also to provide young professionals an opportunity to experience at close quarters development work in India? Both are objectives of the IP but it is important to understand which objective takes precedence. Nurturing young professionals for development work with children is an important mandate for internationally renowned organizations like UNICEF. One of the objectives is providing interns an opportunity to learn in a multi-cultural multi disciplinary environment and to bring a fresh perspective to ongoing social problems. If objectives are revised, strategies to achieve them will also need to be identified and a theory of change developed to understand the pathways to achieve these objectives.

Recommendation Three: UNICEF ICO and state offices to brand and advocate the IP nationally and internationally

47

If UNICEF leaders at ICO and the state become IP champions, they could energize and increase the value of the IP in India. Internationally, the unique design and brand of the Indian IP – the selection process, multi-cultural teams of interns, the collaboration with research organizations at state level, the field work opportunity and a concrete KM product vetted for quality – needs be disseminated and promoted.

Programming Issues Lessons learned in the area of effectiveness were that 1) many host organization had limited research capacities. Research capability and inputs by all stakeholders in the IP is important – interns, UNICEF ICO, state offices and host organization – to produce a quality document. The interns are highly qualified, selected rigorously and able and willing to engage in the research process. If the IP product is to be positioned as a KM product involving a research topic that is rigorous in terms of research design and data, then it must be backstopped with personnel who are able to provide adequate inputs – UNICEF ICO, state office and the host organization. The emphasis should not be on who will host interns but rather, what the focus of study is and then decide who is the best host 2) government and host organizations appear to have limited involvement in the TOR development for interns. However, only demand driven documentation will ensure both ownership and use. The research topic must be an expressed need by relevant stakeholders especially government and well articulated so that government, host organizations, interns and UNICEF clearly understand what data is to be collected, why and how it is to be used 3) UNICEF depended on host organizations for quality control but there were different expectations regarding the quality of data collection and the report 4) there were missed opportunities for interns, UNICEF and host organizations to work collaboratively and consistently along with government and 5) dissemination and use needs better targeting to influence users and decision makers.

Selection and motivation of the host institution is key to the success of the IP as they will be the mentors for interns and their first point of contact. Based on the topic, the research institute needs to be selected, having relevant expertise in content matter and research skills The selected host organization must have the motivation and commitment to mentor the interns. Clear communication of the host organizations with UNICEF and government is critical since data will be collected in government supported programs and there are multiple expectations of various stakeholders.

Lessons learned from efficiency were that the IP program and development of a KM product was cost effective. Through the division of labor, many tasks had been outsourced and the increasing use of digital dissemination enabled efficiency. The time allocated to the interns, ten weeks, is short, making it difficult to produce a good knowledge product.

The time period of two months is very short for interns to understand and adjust to the context in India, acquaint themselves with their teammates, understand the host institution and state UNICEF office operations, make sense of the TOR, complete timely and quality data collection, analyze data and produce a quality document for dissemination and use. This creates pressure on interns, host organizations and UNICEF to complete the task efficiently and effectively.

The selection of interns has been streamlined and outsourced to a consultant that ensures transparency of selection from a variety of applicants from India and abroad. However, catchment areas from India and internationally have remained the same over the years for various reasons: some educational institutions over the years have streamlined processes to

48

disseminate information to students, interns are likely to promote IP in their own institutions and IP advertisements are sent to the same websites and institutions.

Recommendation Four: UNICEF (ICO and state) to support high quality research skills and provide timely technical backstopping

UNICEF ICO will need to provide technical oversight regarding the research identification, rigor and feasibility (in the two months of time available) as expressed in the TOR. The SPPME is eminently suited for it but the nature of the inputs will be different – more inputs for research and evaluation and less for logistics (which is anyway outsourced to consultants and an event management company). This may require greater engagement of the Research and Evaluation Specialist and a qualitative shift in the type of support provided by the IP Manager to providing inputs that will strengthen the quality of the case studies. UNICEF ICO needs to consider interns recommendations for optional workshops in data analysis, report writing and interviewing prior to the commencement of the IP to strengthen research skills.

The UNICEF state office will need to collaboratively involve the leadership, sector specialist and the SPPME officer, government and relevant host institutions to choose appropriate areas of study and manage the IP. UNICEF state offices need to review the IP program not as a program to host interns for two months but as an opportunity for producing a KM product that will be used using a cost-effective, good quality and fresh-eyed human resource. These systems are in place in many state offices and should be encouraged. If planned well in advance, the intern’s work (preferably beyond print) will substantially contribute to the KM needs. This engagement not only of the SPPME staff but sector (education, health and so on) from the initial stage of identifying the need for research (and not just documentation) supported by the Chief of Staff at state level is critical for raising the profile of the IP program and ensuring quality and usable products.

Hiring consultants to manage the interns at the state level can be an option provided that they have the right skill sets to provide both technical (content and research) inputs, mentoring support and can closely communicate with all stakeholders at the state level and UNICEF ICO.

Recommendation Five: UNICEF to extend the reach of IP recruitment in India and abroad

What is recommended is to increase the multi-cultural component of the interns and widen the net to include universities beyond North America and in India beyond Delhi and Maharashtra. This will require active advocacy and engagement with past interns, identifying gaps in catchment areas and having a plan to reach out to these under-represented locations.

Recommendation Six: All stakeholders to make creative use of the short timeline through early introductions, understanding the context and guidelines for quality documentation

A clearer description of the local conditions in rural India would ensure that interns are emotionally prepared and if not, can opt out before the final selection is made.

Timeline of the IP is two months but the dialogue between the interns, host and UNICEF state offices could begin prior to their arrival. Teams would know each other and get adjusted, they will be introduced early on to the mentor, understand the study topic better, refine TOR and streamline the research process. This will provide a head start to all and may require a rearranging of logistic timelines. The interns under these circumstances will be able to efficiently use the time available for the data collection, analysis and so on. Providing clear guidelines for

49

case study quality early on will streamline the later selection for publication, increase the number of case studies passing the quality criteria and help to reduce the time lag of six months for case studies to be uploaded on the website.

Recommendation Seven: UNICEF to collaborate with key stakeholders to develop good quality TORs

This will require UNICEF ICO to dialogue with the state offices to provide guidelines for developing strong and realistic TORs. The interns’ study must be considered as one piece of documentation or research that fits in with a larger research focus instead of a focus on a subject that may or may not be of value. The expressed need should coincide with UNICEF’s and the government’s priorities. Research topics have included best practice or documentation of UNICEF supported programs (includes government programs). The research area could be expanded to include any area that the government or host institution would like to explore in a formative way such as a needs assessment or contextual analysis of children at risk. In other words, the intern’s focus could be widened to inform a better understanding of the context in which children live or ongoing research and evaluation of UNICEF and the state. The short two months of the IP in such a scenario would not be viewed as a constraint. Once research topics are identified, rigorous TORs in terms of research methodology and sampling can be developed.

Operational Issues

Overall lessons learned are that there is inter-connectedness with many stakeholders and that the role of each of the stakeholders is important to achieve the outcome of knowledge generation and use in order for there to be sustainable results and achieve impact.

The IP is currently viewed as a UNICEF ICO program driven by SPPME, although participation of state UNICEF offices is voluntary and the IP is decentralized in operation. Dilemmas relating to how the IP will benefit UNICEF state offices, why the investment in time and human resources is necessary, why it is important to collaborate with government and host organizations at a strategic level and other such issues affect the ownership and commitment to the IP.

UNICEF has not engaged actively in the alumni network created over many years missing out an opportunity to create advocates and build knowledge and partnership networks. UNICEF posting some information occasionally on the over 300 strong Facebook page is only a small beginning.

Recommendation Eight: UNICEF ICO and state offices to be proactive in identifying and building relationships with quality research institutes to support the IP as well as other research needs.

One of the KI mentioned that the good research institutes would engage with UNICEF only if they see value in the relationship. UNICEF ICO and state will have to reach out to such institutions, develop MOU not just for IP but in the long-term to advance KM for children in India. Short-term agreements do not benefit either UNICEF or the host organization. The point person or mentor must be committed to the program (and not changed or be unavailable at the last minute) and be willing to be challenged, to learn from interns as well as help them learn. If the host institution and mentor are able to utilize IP for their own academic or work related advancement such as using the data for other studies or to write a paper or build up their own

50

teaching materials, the relationship with interns will be intellectually stimulating. Host institution selection is based on two requirements one is related to the research capacities and the other is the ability to house the interns with available infrastructure. The latter should not be a constraint in selection of the host institution. To enable deeper engagement of the host organization, their involvement in the development of the TOR, engagement with pre-formed teams regarding the scope of work prior to their arrival and discussion on the use of the product would make the process meaningful. Many states are constrained with the quality of the host institutions available – in this case UNICEF state offices may need to step up their involvement to ensure the quality of the research produced by interns.

Recommendation Nine: UNICEF ICO and states to develop/strengthen strategies for advocacy and use of case studies (and other knowledge products) with government

Advocacy for use by UNICEF key decision makers are likely to have weightage to encourage government to use case studies and related knowledge products for program design, improvement and to inform policy. Various recommendations stated earlier such as engaging the government in development of the TOR and disseminating findings will support this recommendation. However, this is not enough to ensure use. Strategies for advocacy and use need to be developed for multiple levels of government – local, state and national. For national level advocacy and use, the case studies will need to be collated with other information available to confidently advocate their findings. This means not only a larger role of UNICEF at ICO and state level to advocate for use but also identifying persons within UNICEF to do so.

Recommendation Ten: UNICEF ICO to strengthen state level ownership for UNICEF and other stakeholders

Ownership of the IP needs strengthening at UNICEF state offices, host organization and the local government. Only when the benefit can be perceived at the state level and the objectives of the IP understood will the ownership of the IP shift. One method of doing so is the constructive use of feedback received every year from the interns. Similarly, UNICEF state offices, host and government should also provide feedback to interns and ICO every year. This will lead to a transparent discussion by all stakeholders – UNICEF ICO, state offices and host – to inform and improve the next year’s IP.

Recommendation Eleven: UNICEF to actively build alumni networks

Building the intern alumni network is recommended and to go beyond the simple set up of a Facebook page. UNICEF can inform and engage past interns at many levels – for the ongoing internship program as virtual mentors, for work related opportunities and to communicate UNICEF’s resources, products, newsletters and plans. Similarly, interns by engaging with UNICEF and host organizations can inform about their work and institutions and provide a pathway for collaboration or exchange of information. Some simple doable suggestions are revamping the KICC website so that the search functions for year, topics, states and authors are easily available, strengthening key word search functions so that googling can throw up the KM products of KCCI and ensuring that use of the case study is reported. A LinkedIn group can encourage interns’ professional networking and enable UNICEF to leverage the interns’ current work related linkages. Social media like Twitter can increase communication between and with interns. An area not explored is the communication and linkages between UNICEF ICO, UNICEF HQ and the academic institutions where the interns are studying. UNICEF needs to capitalize on its young ambassadors now 500 strong and reach out and continue the process of KM started so many years ago.

51

Attachment One: Terms of Reference

TERMS OF REFERENCE

Evaluation of KCCI Internship Programme

30 April 2012

Study (an investigation designed to improve knowledge on a particular topic)

Survey (an assessment of the conditions of a particular group at a point in time)

Evaluation (an assessment of an on-going or completed project, programme or policy)

Background

The Knowledge Community on Children in India (KCCI) initiative aims “to ensure that relevant knowledge and leading research is brought to bear on policies and programmes for children in the country”. KCCI is “a community that promotes information sharing on policies and programmes related to children in India”. KCCI comprises the following:

1. Summer Internship Programme 2. Publications 3. Website 4. Resource Centre

KCCI was launched in 2005 as one of UNICEF India’s (ICO) first concerted knowledge management (KM) efforts. Since that time, the importance of KM to UNICEF’s work in India has become increasingly recognised, such that one of the key strategies for achieving the ambitious results of the 2008-12 GOI-UNICEF Country Programme is “providing technical assistance and support to improve knowledge management systems and sharing of lessons learned”

A very vital component of the KCCI initiative is Summer Internship Programme (IP) which has been functional since 2005.

Between 40-100 graduate students from India and abroad participate in a ten-week long IP. Each team of interns comprises a combination of Indian and international students who write a case study on a project relating to child rights. The interns are placed with a research or implementing partner who supervises their work, with guidance from UNICEF both at state and national levels.

52

Since 2005, 494 interns from India and abroad have participated in the programme. They have been hosted by 70 research/academic institutions across the states. They have worked in all states where UNICEF has presence except Chhattisgarh. Interns have documented 129 case studies out of which 88 have been published after being reviewed by the field offices and programme sections.

The original - objectives of the IP as formulated in 2005 were to: a) Produce high quality documentation of innovations in service delivery, what

works (and what does not), why, and under what conditions, so as to use this to influence and inform the development discourse and disseminate evidence based research amongst government functionaries, development practitioners and policy-makers

b) Collect field-work based data and information relating to children across India which will offer comparable snapshots into progress made in selected issues relating to children’s development, as well as complement UNICEF’s documentation needs.

c) Support and develop a cadre of young research and development professionals with interest, commitment and skills relating to children’s development in India

d) Develop partnerships with leading research institutes in India and promote greater interest in social development research.

e) Promote awareness about UNICEF activities among graduate students in India and abroad

The assessments completed by the interns every year have consistently indicated that over 90 per cent of the past participants were satisfied with the programme. The anecdotal evidence suggests that it has made progress towards the objectives: Considerable number of the past interns continues working in areas related to children’s development and studies conducted by interns have contributed to building the knowledgebase on issues related to child rights.

At the same time, a question has been raised as to whether the Internship Programme is an appropriate approach to knowledge management for UNICEF India which continues to move upstream and is increasingly expected to play a role as a policy organization. Furthermore, the Internship Programme is an intensive programme that requires significant financial, human and technical resources to manage the programme.

UNICEF India will enter into a new five-year Country Programme from 2013. Knowledge management will remain as one of the key strategies and as an intermediate result that reads, Evidence-based strategic knowledge and advocacy tools on prioritized issues related to child rights developed, documented and disseminated. This intermediate result is intended to contribute to the Programme Component Result 4: Policies, practices, programmes, public opinion and social norms advance the rights of children, adolescents and women.

53

Rationale for the Research Activity

The KCCI IP is into its eighth year In the last 7 years, between 2005 -2011, but the programme has never been evaluated. As UNICEF India moves into next planning cycle -2013-2017- an evaluation is proposed to assess if the original objectives of the Internship Programme have been met and to what extent it has contributed to the knowledge management results of UNICEF India. It is intended to gain evidence to decide upon the future of the Internship Programme.

Use of the findings

The findings will inform the future plan and design of the Internship Programme.

As the Internship Programme is an integral part of the Knowledge Management strategy of UNICEF ICO, the evaluation finding can also help refine the KM strategy for new country programme.

Scope of the Research Activity

The research needs to look into the Internship Programme in the current Country Programme (2008 – 2012). The evaluation will overlap with the current year’s programme so that the consultant is able to observe the opening and closing workshop and discuss with interns and other stakeholders.

To answer if the original objectives of the Internship Programme have been met and to what extent it has contributed to the knowledge management results of UNICEF India, the Programme will be evaluated as per the following questions and criteria: .

A. How relevant has the internship been?

i) How relevant is the programme in relation to UNICEF India’s knowledge management results (as per the County Programme strategies and Intermediate

Results)? ii) To what extent will the objectives of the programme be still valid

in the context of the new Country Programme (IR 62 under PCR 4)

iii) How relevant has Internship Programme been in terms of meeting the knowledge demand/needs of other IRs

iv) Are the structure and design of the programme appropriate to achieve the objectives of the programme? Are they appropriate to achieve the knowledge management results?

a) Intended output, duration, administration, management mechanism b) Interns (selection, composition, etc) c) Selection of topics and research institutes

54

B. How effective is KCCI IP? i) To what extent has the Internship Programme met its original

objectives? ii) What evidence is there to support the achievement or non-

achievement of objectives? iii) What are the perceptions of UNICEF and interns? iv) What is the quality16 of the internship case studies and papers

published since 2008? a) Are the case studies consistent with the terms of reference? b) Do the case studies have an executive summary that

summarizes the key findings, conclusion and recommendations?

c) Do the case studies have actionable recommendations supported by evidence (corresponding to findings and conclusion)?

d) What are the perceptions (in terms of quality) of UNICEF POs, research institutes, government counterpart, interns?

e) Do the terms of reference explicitly state the intention or specific plans to use the findings for strategic decisions?

C. How efficient is KCCI IP? i) What has the cost been of implementing KCCI IP to date (financial, human

resources)? ii) Did KCCI use its resources in the most economical manner to achieve its

aims? iii) Are the case studies disseminated in a systematized and efficient way?

D. How sustainable is KCCI IP? i) What scope is there for KCCI IP to be funded /run by MWCD or other

relevant institutions (e.g. academic institutions)? ii) What elements of risk are there to this programme and how have they been

articulated and/or addressed?

E. What kind of impact of KCCI IP can be ascertained to date? i) To what extent have previous interns continued working on issues related to

children or human rights and development in general17? ii) What evidence is there that the evidence brought out by the case studies

have been used by UNICEF, government counterparts or others to inform programme design?

iii) Have there been any spillover effects e.g. is the internship programme led to the creation of knowledge networks with research institutions/universities that

16

Benchmarks to assess “quality” may need to be established, whether through an assessment against similar products, or through a review of internship case studies themselves 17

In this context, the evaluator will need to capture a representative/differentiated sample of the seven batches of interns to date, including what they are doing now

55

have benefited the office as a whole? For example, the identification of new research agencies that could be approached for other pieces of research/documentation work?

iv) To what extent has UNICEF ICO tapped into the network created by interns? v) Did you learn about any UNICEF activities during the IP that you were not

aware of before? What more did you learn about activities that you were already aware of?

The conclusion should include what would UNICEF India gain/lose by not having KCCI.Particular emphasis should be also paid to actionable recommendations for continuing/stopping/improving KCCI Internship Programme.

Methodology

The evaluation design will use a mixed methods approach combining quantitative and qualitative tools and analysis. The methods used will be: document review (including products and materials developed by interns), questionnaire, interviews with key stakeholders (UNICEF and intern hosting institutions) as well as interviews and focus group discussions with interns.

For interviews and FGDs, a purposeful sampling of interns (national and international) is proposed to ensure the least bias and maximum coverage.

Sampling:

There are two default strata of the erstwhile KCCI Interns –A. National and B. International;

There is another natural strata which can be taken as the ‘year of interning with KCCI’. We can make two strata based on year of attendance as

2008-2009

2010-2011

30% sample is suggested keeping a provision of about 30% non-response within the sample (or overall 9 to 10% non-response).

Sample to be proportionally distributed in the 4 categories:

a) National – 2008-2009,

b) National – 2010-2011

c) International 2008-2009

d) International 2010-2011.

56

The spread will ensure representation of all types of interns from different years.

Research Question Methodology

A i, ii

B ii, iv a,b, c, e

C i, ii

D ii

Desk review of existing documentation; UNICEF MTSP, NYHQ Knowledge Management concept note, CPD, CPAP, KM in ICO concept note, KCCI strategy paper and Interns evaluation (2008-2011); Terms of Reference for the study and case studies and papers produced by interns, Information on financial expenditure

A i, ii, iii

B iv d

B i, ii, iii

C ii, iii

D i

E ii, iii, iv,

Key Informant interviews with UNICEF staff (Senior management, CFOs, Chiefs of Programmes,and PO’s who have been involved with the programme), counterparts, host institutions, universities

B ii, iii, , v

C ii, iii

E i, iv

Email survey18

Schedule of Tasks & Timeline

Task Anticipated Timeline (e.g. # of weeks)

Desk Review 2 weeks

Key Informant Interviews/FGDs 2 weeks

Email survey of interns 1 week

18

More than 50% of the interns are part of Facebook Group. Response of this group can also be solicited.

57

Write report 1 week

Estimated duration of contract 6 weeks between May and August

Deliverables Inception report with evaluation plan 1 evaluation report of between 40 – 50 pages (not including Annexes) following

UNICEF Report writing guidelines (to be provided). 1 powerpoint presentation summarizing key findings and recommendations

Qualifications & Experience required

It is assumed that the assignment can be done by one consultant or a team of a few consultants.

While the evaluation questions are listed in detail below, it is proposed that only the primary evaluation question – purpose of the evaluation – and the key question under each criterion are shared with the consultants. They will be asked to unpack the evaluation questions based on the purpose and use of findings. Proposed evaluation questions will be reviewed to assess the technical capacity of the agencies/consultants.)

Proven experience in conducting evaluations. Experience in knowledge management is preferable.

Advanced university degree in Social Sciences or Development fields Strong evidence of good writing skills in English Ability to write in plain language style. Ability to work independently Reliable internet connectivity.

The consultant(s) are expected to abide by the United Nations Evaluation Group Norms and Standards.

58

Attachment Two: Short Term Contract of host organization

Small Scale Funding Support Agreement

Based on the Country Programme Action Plan between the Government of India and UNICEF, UNICEF agrees to co-operate with National Commission for Protection of Child Rights (henceforth, NCPCR) as described below.

1. Work Plan Results to which the small-scale support contributes to (convergent) IR IR62, Strategic knowledge on MDGs, social inclusion, equity and child rights developed, documented and disseminated to inform programming and policies for multiple audiences.:

The current agreement is towards collaborating on Knowledge Community on Children in India (KCCI) summer internship programme. The internship programme aims to strengthen and promote knowledge sharing on children’s issues by documenting selected intervention, in the form of written outputs.

2. Activity or activities to be carried out with the support of the small-scale support:

NCPCR agrees to be a Partner in the implementation of the Programme, and complete the following tasks, in conformity with the Programme budget as allocated by UNICEF:

(a) Host 1 group of 4 interns and provide day to day supervision to the interns (both desk review and

field research) and provide quality assurance of the documentation under the agreed framework. Work plan to be agreed with supervisors and students;

(b) Provide desk space, miscellaneous consumables such as telephone communication, computers, internet connectivity and stationary to the interns;

(c) Assist the interns in finding accommodation provided however that the interns shall be responsible for the cost of such accommodation;

(d) Where relevant to the documentation, coordinate field trips, including visits to the field site, arrange for translator/interpreters services, and transport and accommodation;

(e) NCPCR will facilitate the dissemination of findings from the output through forms of workshops/meetings at the state/institution level

(f) NCPCR agrees to brand the written output along with UNICEF and other partners of Knowledge Community on Children in India

(g) While the interns will be covered by their own medical insurance and therefore there will be no financial liability on the partner on this account, NCPCR will assist interns in accessing medical care in case of any medical emergency

(h) In case the assignment involves behavioral research, NCPCR will get the research design and questionnaire approved by a review body (consisting appropriate mix of internal and external members) constituted for this purpose, with the aim to protect the rights and welfare of the research subjects

3. Expected results/outputs to be achieved:

A publishable Good Practices/Lessons Learned on Documentation and analysis of complaints received by the National Commission for Protection of Child Rights and response mechanism for addressing these complaints

4. Starting and ending dates for implementation of the activities:

25 May to 3 August 2011

5. UNICEF will contribute the following resources:

59

a) Funds: INR 305,000 (details given below)

Payment to the Partner of Rs 150,000 for each group of 4 interns, to cover the costs of lease of computers, printers, faculty charges, internet access, postage and phone bills

Reimbursement of actual travel costs including train tickets, accommodation, transportation, resource person, and translation services of Rs 75,000 per group of 4 interns (to be provided on actual cost incurred)

UNICEF will provide Rs 80,000 for four interns to cover a period of two months. NCPCRwill make the payment to the interns at the rate of 20,000 per month.

b) Supplies: Not Applicable

c) Technical assistance:

Identifying the interns and intern groups for whom the Partner shall be responsible;

Providing the Partner with the names and any other relevant details of the interns at least 2 weeks prior to their arrival as scheduled for the period 25 May to 3 August 2011;

Providing orientation and training to the interns in May 2011 and prior to their dispatch to their joining NCPCR

d) Other inputs: Technical guidance and support through staff time

6. The NCPCR will contribute the following matching resources:

a) Funds : Nil

b) Staff/people : NCPCR will designate a person who will be given authority for overseeing the Programme on its behalf and act as focal point in all communications with UNICEF. The person will participate in orientation for interns and meeting of partner institutions in Delhi on 27 May 2011 and closing workshop on 2 or 3 August

c) Equipment/in kind: As per the agreed budget

d) Other : As per requirements

7. (If applicable) Portions of the small-scale support will be transferred by UNICEF in the following stages:

Amount/Date

UNICEF, Delhi will provide 50% of the advance to NCPCR within two weeks after both UNICEF and NCPCR have signed the agreement. The remaining amount will be transferred on submission of final report and utilization certificate

8. The Organization will provide the following reports at or near the given dates:

Reports Date

Narrative A final report on the outcome of the programme, within a month of the completion of the programme

Financial Statement of accounts and utilization certificate after completion of the agreement period

9. The resources provided by UNICEF will only be used by NCPCR in pursuit of the project/sub-project objectives, for the activities to achieve the results as agreed to in 1 – 3 above.

10. The attached Project Proposal and Budget are part of this agreement.

60

11. UNICEF and the NCPCR will cooperate to monitor the results of this project

12. The Organization may only use the UNICEF name, logo and emblem in connection with the Project with the prior written consent of UNICEF.

13. This agreement can only be changed through an agreed modification in writing.

14. Place and date

Currency: INR

61

Attachment Three: Evaluation Matrix

a. Relevance (Questions from TOR)

How relevant is the program in relation to UNICEF India’s knowledge management results (as per Country Program strategies and IR)?

To what extent will the objectives of the program be still valid in context of new country program (IR 6.2 under PCR 4)

How relevant has internship program been in terms of meeting the knowledge/demands of other IRs? Relevant Outcomes Relevant IP

outputs/activities Indicators Data Sources

IP is consistent with KM results as indicated in the planning documents

Five year plans 2008-12, Concept papers and other guidance materials on IP and KCCI

Other documents, as relevant

Analysis of the objectives, principles, and initiative components in key IP KCCI documents

Desk review of key documents

Degree of sentiment among most relevant stakeholders (e.g., UNICEF ICO/unit heads/state staff, thought leaders, etc.) that IP initiative outputs are aligned with knowledge management results

Interviews (by inclusion of alignment-related questions in stakeholder interview protocols)

The question ‘to what extent will objectives of program be valid in the new country program’ is part of the TOR but subsequent discussions inform that as the new country program is under development, it would not be possible to address it.

Are the structure and design of the program appropriate to achieve the objectives of the program? Are they appropriate to achieve the knowledge management results? Intended output, Duration, Administration, Management mechanism; Interns Selection and Composition; Selection of topics and research institutes

Relevant Outcomes Relevant IP Activities/Outputs

Indicators Data Sources

Interns, host institutions, UNICEF ICO and state staff, relevant key government and other development actors and other

Intern selection criteria

TOR selection

Case study guidelines

Case studies published

Roles and responsibilities of

Level of self-reported stakeholder understanding (including UNICEF staff members themselves) of the IP process

Survey (by inclusion of relevance-related questions)

Level of demonstrated stakeholder evidence of quality, use of published case studies and how it

Interviews with ICO, host organizations, govt. officials, interns

Review of existing

62

key stakeholders understand IP initiative objectives and the means to achieving them

partners and UNICEF ICO and state

Intern feedback

Partner feedback

was relevant to their work

evaluation reports

Documentation (Case studies, TORs)

b. Effectiveness (from TOR)

To what extent has IP met its original objectives?

What evidence is there to support the achievement or non-achievement of objectives?

What are the perceptions of UNICEF and interns?

What is quality of internship case studies and papers published since 2008?

Are case studies consistent with TOR?

Do case studies have an executive summary that summarizes the key findings, conclusion and recommendations?

Do case studies have actionable recommendations supported by evidence? (Corresponding to findings and conclusion)

Perceptions of quality of UNICEF POs, research institutes, govt. counterpart, interns?

Do TOR explicitly state the intention of specific plans to use the findings for strategic decisions?

Relevant Outcomes Relevant Initiative Outputs


Interns, UNICEF ICO and other key stakeholders are aware of the IP’s main outputs, disseminate, advocate and use them

Partner organizations and govt. expand work on children’s issues and forge partnerships with UNICEF

Interns continue to work with children or development issues

Case studies

Supporting

and other guidance materials (TORs, dissemination plan)

Extent interns, UNICEF ICO and other key stakeholders are aware of initiative’s main outputs and use them

Extent of key stakeholders demonstrating the knowledge that key initiative materials (case studies) to which they have been exposed have sought to convey

Percentage of published case studies (having been approved through external reviewers)

Extent of interns and other key stakeholders demonstrating support for initiative and the ways in which UNICEF has contributed to this process

Desk review of M&E data generated, and evaluations produced, to date

Review of case studies

Interviews (for perceptions of awareness, effect and support gained; skills gained and of use)

Relevant documents for evidence (use of case study)

63

Communication from and with interns

Increase in Interns engagement in development issues especially children

Extent of interns reporting interest in development issues especially related to children (e.g., through jobs, further education, sharing of materials)

Interviews/surveys with interns and key stakeholders

c. Efficiency

What has the cost been of implementing KCCI IP to date (financial, human resources)? Did KCCI use its resources in the most economical manner to achieve its aims? Are case studies disseminated in a systematized and efficient way?

Relevant Outcomes Relevant Initiative Outputs


Case studies are produced by interns and published by ICO in a timely fashion

UNICEF makes the best use of the internal and external resources at its disposal to produce key initiative outputs such as selection of interns, TORs, management of interns and case studies production

The costs associated with producing and disseminating case studies justify the benefits emanating from these outputs

Case studies Each key initiative output is timely

Internal skills and expertise within UNICEF (CO and state staff) are strategically identified and deployed to each key initiative output

External skills and expertise (partners, consultants) are strategically identified and deployed to each key initiative output

Ratio of input units (human and financial resources, including monetized staff costs and opportunity costs to other work) to the outcome units produced by key initiative output i.e. case studies. A unit cost will be calculated for each year

Desk review of budget broken down by year

Interviews with staff, partners, and others

Informal analysis of budget data alongside evaluator’s professional judgment of the overall effectiveness of output

d. Sustainability

What scope is there for KCCI IP to be funded or run by MWCD or other relevant institutions (e.g. academic institutions?)

What elements of risk are there to this program and how have they been articulated and or addressed?

Relevant Outcomes Relevant IP Initiative Outputs


64

The benefits emanating from key initiative outputs are poised to outlast UNICEF’s direct involvement in producing these outputs

Case study Willingness of others to fund the program

Interviews / Focus groups with ministerial officials, UNICEF administrators, partners and other thought leaders

Sustainability i.e. would Women and Child Department of Govt. of India support the IP is a policy level question and cannot be answered by this evaluation. For this reason the indicator is rephrased as ‘willingness’ to fund the program.

e. Impact To what extent have previous interns continued working on issues related to children, or human rights and development?

What evidence is there that the case studies evidence has been used by UNICEF, government counterparts or others to inform program design?

Have there been any spillover effects e.g. IP led to the creation of knowledge networks with research institutions/universities that have benefited the office as a whole? For example, in the identification of new research agencies that could be approached for other pieces of research/documentation work?

To what extent has UNICEF ICO tapped into the network created by interns? Questions would relate to understanding how ICO maintains communication with past interns to promote, share and utilize available expertise.

What have interns learned of UNICEF activities during the IP that they were not aware of before? What more did they learn about activities that they were already aware of?

What would UNICEF India gain or lose by not having KCCI?

Relevant Outcomes Relevant IP Initiative Outputs


Increased education or job selection in development preferably working with children attributed to the internship

Increased use of case studies for program and policy

Case study

Degree to which interns report increased education or job selection in development

Degree to which partners, UNICEF, govt. report use of case studies

Degree to which UNICEF, interns, partners indicate communication for professional development, networking and knowledge increase about children and UNICEF activities

Interviews (for anecdotal evidence speaking to impact)

Survey of interns

65

Attachment Four: Theory of Change

These interventions or activities included in the Theory of Change are as follows:

1. Selection of case study issue through internal meetings with sectors, CPO, state

management meetings (SMT) 2. Joint meetings with Govt. to select case study issue based on criteria (available work

plan, other) 3. Debrief meetings with govt., partners and UNICEF 4. Meetings to plan what next after case study for joint or own research, evaluation or

program 5. Set up reviews for case studies – criteria and who will do it 6. Develop dissemination and use plan by each stakeholder 7. Develop MOU; match with interns 8. Hold regular meetings with interns 9. Review case studies drafts periodically 10. Announcement for internship in countries, universities, virtual; selection criteria 11. Training program at UNICEF ICO and State level 12. Develop case study quality standards; feedback to interns 13. Develop sharing platform for interns, UNICEF, others to communicate professional

plans, information, etc e.g. Facebook to know updates 14. Budget for planned and piggybacking on other meetings; print case studies; update

website; track use

The assumptions of the TOC are:

1. UNICEF has the resources – human and financial, capacity and sector expertise to run

the IP program. 2. Partner organizations are available and willing to participate 3. Interns want to come for the UNICEF internship program. 4. The team pairing of national and international – is useful for both 5. Case studies that are rigorously reviewed will be used 6. The right candidates are selected as interns

66

Attachment Five: Relevance of IP Program: Related Tables

Perception of interns about relevance of objectives

7. The following are the objectives of the Internship Program. Looking back, how relevant do you think they are


Very relevant Relevant Not much

1 2 3

Produce high quality documentation (case studies) to influence govt., policy makers and development practitioners

32 27 8

48% 40% 12%

Collect field work data on selected issues and complement UNICEF’s documentation needs

45 20 2

67% 30% 3%

Develop cadre of young professionals to understand children, development in India

43 18 6

64% 27% 9%

Develop partnerships with leading research institutes and promote further interest in development research

26 32 9

39% 48% 13%

Promote awareness of UNICEF among students in India and abroad

32 24 11

48% 36% 16%

Perception of UNICEF managers about relevance of objectives

6. The following are the objectives of the Internship Program. Looking back, over four years 2008-2011, how relevant do you think it is?



1 2 3


2 9 5

12% 56% 31%


5 9 2

31% 56% 12%


11 5 0

69% 31% 0%


5 9 2

31% 56% 12%


9 6 1

56% 38% 6%

67

Perception of host organizations about relevance of objectives

6. The following are the objectives of the Internship Program. Looking back, over four years 2008-2011, how relevant do you think it is



1 2 3


7 5 3

47% 33% 20%


8 6 1

53% 40% 7%


7 6 2

47% 40% 13%


11 3 1

73% 20% 7%


9 5 1

60% 33% 7%

68

Attachment Six: Effectiveness of IP program: Related Tables

Interns work on a project that has been identified as important by and for government. UNICEF with support from others develops the Terms of Reference (TOR). How can TOR Development be strengthened?



Host organizations

Yes Yes Yes

1. The research topic must be very specific in scope of work and expectations from interns

65 16 14

97% 100% 93%

2. Research must involve on the ground field research and not secondary data collection

56 14 12

84% 88% 80%

. Host organization and field organizations must have adequate capacity to mentor interns

67 16 14

100% 100% 93%

. TOR must have good research and sampling design 60 16 11

91% 100% 79%

. TOR must NOT be developed by host organization alone. UNICEF and others such as target community and government must be involved

59 16 11

89% 100% 73%

How can the quality of case studies produced by interns be improved?



Host organizations

Yes Yes Yes

1. Decision for case study publication should be made in a month, soon after internship ends

44 13 10

67% 81% 67%

2. Optional workshops are needed during the internship orientation (Delhi) on a variety of topics such as writing skills, GIS mapping, interviewing techniques, data visualization, data analysis

53 10 10

82% 62% 67%

3. Increase the time for the research from 2.5 months to 3.5 36 8 7

55% 50% 47%

4. Strengthen research design and sampling 64 16 12

97% 100% 80%

5. Data analysis support from a statistician or other technical person needed during internship

50 13 14

76% 81% 93%

6. Feedback (proofreading and input) on draft case study from host institution and sharing of all data available

52 15 13

81% 94% 87%

7. Feedback (proofreading and input) on draft case study from local UNICEF office is required

57 15 11

86% 94% 73%

8. UNICEF to regularly follow up with interns on research progress to ensure quality of data collection

61 15 10

94% 94% 67%

69

Interns’ Perceptions: Please explain their contribution and/or reaction regarding quality and use (referring to above question)


UNICEF Host Organization Govt.

1 2 3

1. Promised to utilize findings 14 3 10

52% 11% 37%

2. Interaction was very brief, so reaction limited 6 7 9

27% 32% 41%

3. Project was scaled up/continued/stopped before the findings were available so not sure if they were useful

7 5 2

50% 36% 14%

4. Findings critically assessed and appreciated 15 4 7

58% 15% 27%

UNICEF State Managers: Please explain their contribution and/or reaction regarding quality and use (referring to above question)



1 2 3


50% 25% 25%


60% 20% 20%


2 1 2

40% 20% 40%


33% 33% 33%

Host organizations: Please explain their contribution and/or reaction regarding quality and use (referring to above question)



70

1 2 3


67% 33% 0%


12% 50% 38%


1 5 0

17% 83% 0%


75% 25% 0%

71

Attachment Seven: Possible impact of IP: Related tables

Looking back, how did the internship programme benefit you?


Yes No

1 2

I was better prepared to enter the development field 58 9

87% 13%

I was able to network and make valuable contacts 41 26

61% 39%

I have a better understanding of the development field in India

64 3

96% 4%

I understood UNICEF’s work better 55 12

82% 18%

Internship with UNICEF opened doors for my academic and/work opportunities

47 20

70% 30%

I learned technical skills such as how to design, implement a research study of high quality

47 20

70% 30%

Encouraged me to start and/or continue work in India 48 19

72% 28%

Strengthened my CV 62 5

93% 7%

To whom did you disseminate the published or final case study? (Do not include the Delhi workshop at end of internship program)



Host organizations

Yes Yes Yes

1. Indian Government officials 18 10 5

33% 77% 38%

2. International government officials 6 1 1

11% 17% 10%

3. Employers – current and potential 27 NA NA

47%

4. NGOs, CBOs, target community NA 11 8

92% 57%

5. Educational institutions – national and international 24 7 7

40% 78% 54%

6. Seminars – national and international 11 6 4

20% 67% 40%

6. Friends, fellow interns and colleagues 43 NA NA

72

72%

7. Others... 14 NA NA

34%

Explain how and why you can strengthen dissemination and use

Looking back, has the Internship Programme contributed to


Yes Yes

UNICEF Managers

Host

UNICEF ICO knowledge building? 13 13

81% 87%

The project’s (case study) knowledge building? 15 12

94% 80%

Government’s knowledge building? 6 7

38% 47%



Host organizations

Yes Yes Yes

1. Search engine optimization so that academic research searches on similar topic leads to KCCI case studies on first Google page; only needs tweaking such as PDF files name is same as case study title.

61 14 13

97% 88% 93%

2. Disseminate through online magazines and social media such as Facebook and Twitter so KCCI reaches out to a large audience and contributes towards development discourse

60 12 11

95% 80% 73%

3. Include a strategy for dissemination in TOR for interns and other stakeholders

56 14 12

88% 93% 80%

4. Provide platform for interns to present their findings on return to parent Indian or international institutions; formally connect these Indian and International institutions

58 12 13

92% 86% 87%

5. Improve KCCI website so that all case studies are available, they open and ensure that they are quickly uploaded for dissemination.

60 16 14

98% 100% 100%

6. UNICEF to inform interns through newsletter or other means how case studies have been used

58 11 11

95% 73% 79%

7. Disseminate executive summary to key stakeholders such as government.

62 15 13

97% 94% 93%

8. Have a presentation of case study at state level for government officials

61 16 12

97% 100% 80%

73

Host organization’s knowledge building? 9 13

56% 87%

Strengthen collaboration with government? 4 4

25% 27%

Strengthen collaboration with host organization? 9 14

56% 93%

How can the Internship Program be strengthened?


Yes No

1 2

One of the criteria for host institution selection must include ’ availability of theoretical and practical research skills’

56 6

90% 10%

Interview the interns before selection to gauge commitment 35 26

57% 43%

Team composition should be reviewed carefully 56 5

92% 8%

There should be clear communication to interns why they are assigned to which project and location

49 12

80% 20%

Team members should sign a bond not to leave early 36 25

59% 41%

Strengthen alumni network 56 5

92% 8%

74

Attachment Eight

Interns are committed to work for children or development or in a developing country

Senior UNICEF staff, partner org., govt. officials, other development organizations, interns are aware of, disseminate, advocate and use case studies

Partner organizations and govt. expand work on children’s issues and/or strengthen partnerships with UNICEF

Interns understand key development issues and bring a fresh and diverse

perspective

IP experience and evidence used to promote the rights of children

Studies produced by interns are of

high quality Case studies are collaborative and

accepted by various

stakeholders

ToRs are well developed and reflect issues important to govt. and/orUNICEF

Interested candidates know of, apply in large numbers

Interns selected with right combination of research, academic, practical experience

Partners have expertise in subject area and

support interns

UNICEF able to identify and work with partner organizations

UNICEF sections or state offices able to identify key issues and are engaged

and havecapacity

Government engaged in selection of topics

Diagram: Theory of Change Internship Program

Suggested TOC

Interns are committed to work for children or development or in a developing country

Senior UNICEF staff, partner org., govt. officials, other development organizations, interns are aware of, disseminate, advocate and use case studies

Partner organizations and govt. expand work on children’s issues and/or strengthen partnerships with UNICEF

Interns understand key development issues and bring a fresh and diverse

perspective

IP experience and evidence used to promote the rights of children

Studies produced by interns are of

high quality Case studies are collaborative and

accepted by various

stakeholders

ToRs are well developed and reflect issues important to govt. and/or UNICEF

Interested candidates know of, apply in large numbers

Interns selected with right combination of research, academic, practical experience

Partners have expertise in subject area and

support interns

UNICEF able to identify and work with partner organizations

UNICEF sections or state offices able to identify key issues and are engaged

and have capacity

Government engaged in selection of topics

evaluation of the unicef india summer internship

Documents