National Institute of Agricultural TechnologyNational Institute of Agricultural Technology

Arnozis, Patricia (INTA)

Bedascarrasbure, Enrique (INTA)

Mirassou, Susana (INTA)

Monti, Eliseo (INTA)

Portillo, José (INTA)

Trebino, Hernán (INTA)

Evaluation Practice in the Early 21st CenturyEvaluation Practice in the Early 21st Century

27th Annual Conference of the American Evaluation Association27th Annual Conference of the American Evaluation Association

October 14-19, 2013October 14-19, 2013

Washington, DC – USAWashington, DC – USA

Designing an integrated system of evaluation Designing an integrated system of evaluation focusing on innovation with a territorial approach: focusing on innovation with a territorial approach:

the case of INTA the case of INTA ArgentinaArgentina

National Institute of Agricultural Technology National Institute of Agricultural Technology

TopicsTopics

The role of INTA in the National Science & Technology The role of INTA in the National Science & Technology

System: a brief overviewSystem: a brief overview

Some reasons for institutional focus changeSome reasons for institutional focus change

From the theory to practice: Learning from the From the theory to practice: Learning from the

experience: experience: Beekeeping at INTA, a case studyBeekeeping at INTA, a case study

Designing a new M&E system

Final Comments Final Comments

ARGENTINAARGENTINA

3,760,000 km3,760,000 km22

Population~ 42 M Population~ 42 M

Government:Government:RepresentativeRepresentativeRepublicanRepublicanFederalFederal

Language:SpanishLanguage:Spanish

President:President: Cristina Fernández de Cristina Fernández de KirchnerKirchner

A public decentralized institution A public decentralized institution depending on the Ministry of Agriculture, depending on the Ministry of Agriculture, Livestock and Fisheries, with operative and Livestock and Fisheries, with operative and financial independencefinancial independence

To carry out and foster actions focused on the innovation of To carry out and foster actions focused on the innovation of agriculture and livestock, agri-food and agro-industrial agriculture and livestock, agri-food and agro-industrial

sectors contributing integrally to the competitiveness of sectors contributing integrally to the competitiveness of agro-industrial chains, environmental health and the agro-industrial chains, environmental health and the

sustainability of productive systems, social equity and sustainability of productive systems, social equity and territorial developmentterritorial development

MissionMission

National Institute of Agricultural TechnologyNational Institute of Agricultural Technology

Technological innovation for Technological innovation for competitiveness and sustainable territorialcompetitiveness and sustainable territorial

development with social equitydevelopment with social equity(Strategic Institutional Plan)(Strategic Institutional Plan)

Headquarters Headquarters

15 15 Regional CentersRegional Centers

5151 Experimental Stations Experimental Stations

5 Research Centers 5 Research Centers

2121 Research Institutes Research Institutes

343343 Extension Units Extension Units

2 private organizations2 private organizations

ORGANIZATIONORGANIZATION

INTA’s Public Budget, Resources INTA’s Public Budget, Resources

EvolutionEvolution

Source: INTA2003: 164 million ARS2012: 1824.6 million ARS2013: 2162.5 million ARSExchange rate: 5.5 ARS = 1 Dollar

Some reasons for institutional focus changeSome reasons for institutional focus change

Increase in food demand due to population growth and Increase in food demand due to population growth and changes in dietschanges in diets

Agricultural production as energy sourceAgricultural production as energy source

Global Change and Environmental CrisisGlobal Change and Environmental Crisis

National organizational factors: the creation of the National organizational factors: the creation of the Ministries of Agriculture, Livestock and Fisheries and of Ministries of Agriculture, Livestock and Fisheries and of Science, Technology and Productive Innovation Science, Technology and Productive Innovation

Internal organizational factors: a major insight on food Internal organizational factors: a major insight on food industry actors and their needs and the family farming industry actors and their needs and the family farming productionproduction

Innovation is a socio-technical practice where different Innovation is a socio-technical practice where different actions flow together and often proceed in parallel actions flow together and often proceed in parallel

rather than in sequence and disseminate in networksrather than in sequence and disseminate in networks

Territory

Knowledge Knowledge

CreativityCreativity

Technology Technology

PRODUCTS

PROCESSES

METHODS

Transformation

Transformation

From the theory to practice: Learning fromFrom the theory to practice: Learning from the experience the experience

From the theory to practice: Learning fromFrom the theory to practice: Learning from the experience the experience

INTA is aligned with national agricultural public policies INTA is aligned with national agricultural public policies in which both innovation and territorial development in which both innovation and territorial development have been stressed as main interestshave been stressed as main interests

The collective ability to frame things in contextual The collective ability to frame things in contextual terms, permits people to focus their knowledge on terms, permits people to focus their knowledge on value creation and to be able to recognize and solve value creation and to be able to recognize and solve complex problems quickly complex problems quickly

The new portfolio 2013 aims at innovation The new portfolio 2013 aims at innovation processes in complex systems and focuses processes in complex systems and focuses

on territorial developmenton territorial development

Main Processes Involved in the Analysis of Main Processes Involved in the Analysis of Portfolios Portfolios

Previous experiences were thoroughly analyzed (portfolios 2006 and Previous experiences were thoroughly analyzed (portfolios 2006 and 2009) and a major number of relevant actors both from INTA and external 2009) and a major number of relevant actors both from INTA and external public or private organizations were involvedpublic or private organizations were involved

This analysis was performed in order to identify what should be stressed This analysis was performed in order to identify what should be stressed in the quest for a higher impact in local territories, social equity and in the quest for a higher impact in local territories, social equity and environmental sustainabilityenvironmental sustainability

Beekeeping Program Beekeeping Program

The territorial approach has its background in the work of the The territorial approach has its background in the work of the Tucuman Santiago del Estero Regional Center. TTucuman Santiago del Estero Regional Center. The target population is he target population is concentrated on the Small Family Farmingconcentrated on the Small Family Farming

The Beekeeping Program (The Beekeeping Program (PROAPIPROAPI) is an example to test some answers ) is an example to test some answers to the new institutional strategy. It is positioned on the Global Value to the new institutional strategy. It is positioned on the Global Value Chain (GVC) developing a technological path for quality honey Chain (GVC) developing a technological path for quality honey production and the management for associative models named as “a production and the management for associative models named as “a collective mode of innovation”collective mode of innovation”

Multi-institutional R&D team that interacts with project promoters and Multi-institutional R&D team that interacts with project promoters and agents of the Federal Program to Support Sustainable Rural agents of the Federal Program to Support Sustainable Rural Development (Development (ProFederProFeder) articulating with the Regional Centers) articulating with the Regional Centers

R&D team deals with present and future chain competitiveness, using R&D team deals with present and future chain competitiveness, using a prospective method and participatory planning. A regional technical a prospective method and participatory planning. A regional technical network ensures that the innovation process is installed in the network ensures that the innovation process is installed in the territories with equity territories with equity

Working on the territoriesWorking on the territories

SU

PP

OR

T F

OR

TH

E

OR

GA

NIZ

AT

ION

OF

TH

E

GR

OU

PS

IN

TH

E T

ER

RIT

OR

IES

GL

OB

AL

MA

RK

ET

AGENCY QUALITY

MANAGER

CERTIFYING

AGENCY

AAppiiaarryy HHaarrvveesstt EExxttrraaccttiioonn PPaacckkaaggiinngg

gggggg T T T T

Self-inspection Self-inspection

Self-inspection

CCeerrttiiffiieedd PPrroodduucctt

Protocol

High Quality Argentine Honey

PORTFOLIO 2013 PORTFOLIO 2013

15 National Programs and 2 Networks were defined such as 15 National Programs and 2 Networks were defined such as Oilseeds, Cereals, Fruits trees, Forestry, Horticulture, Animal Oilseeds, Cereals, Fruits trees, Forestry, Horticulture, Animal Production and Health, Beekeeping, Natural Resources and Production and Health, Beekeeping, Natural Resources and Environmental Management, Biotechnology, etc.Environmental Management, Biotechnology, etc.

A new set of tools for organizing the proposals was designed A new set of tools for organizing the proposals was designed and the temporal length of the projects has been enhanced and the temporal length of the projects has been enhanced from three years up to six from three years up to six

Platforms –Platforms –Integrators conceived as a managing tool that Integrators conceived as a managing tool that integrates the different strategies carried out by the specific integrates the different strategies carried out by the specific projects in order to achieve contributions or solutions to the projects in order to achieve contributions or solutions to the identified problemsidentified problems

Specific (conventional) Projects, which carry out substantial Specific (conventional) Projects, which carry out substantial activities activities

Regional projects with a territorial focus, regarding geographic Regional projects with a territorial focus, regarding geographic influence area, and attending the producers’ needs within their influence area, and attending the producers’ needs within their contextscontexts

Designing a new M&E system Designing a new M&E system A more adequate and iterative M&E system is being designed A more adequate and iterative M&E system is being designed M&E forming part of the original design process, with a M&E forming part of the original design process, with a

greater focus on learning processes within communities, greater focus on learning processes within communities, stakeholders involved and providing management information stakeholders involved and providing management information for project teamsfor project teams

The outcomes and impacts need greater emphasis and M&E The outcomes and impacts need greater emphasis and M&E linked with planning linked with planning

The accountability to beneficiaries, partners and donors is The accountability to beneficiaries, partners and donors is criticalcritical

Learning is viewed as the ability to constantly improve the efficacy of action

Here we place M&E in a wider context as it relates not Here we place M&E in a wider context as it relates not only to projects but also to programs, organizational only to projects but also to programs, organizational

performance and institutional changeperformance and institutional change

M&E activities at different planning M&E activities at different planning unitsunits

Ex-ante EvaluationEx-ante Evaluation

Cross monitoringCross monitoring

Half -Term EvaluationHalf -Term Evaluation

Final EvaluationFinal Evaluation

M&E activities at M&E activities at different planning different planning

unitsunits

Ex-ante EvaluationEx-ante Evaluation: : to to strengthen strengthen project qualityproject quality

Self-EvaluationSelf-Evaluation

External Peer ReviewExternal Peer Review

M&E activities at M&E activities at different planning different planning

unitsunits

Cross monitoringCross monitoring: : to strengthen project to strengthen project management management

Improvement of management through learning Improvement of management through learning by doing and experience exchange by doing and experience exchange

“ “Internal” and “crossed”: will be done by Internal” and “crossed”: will be done by members of the same program and from related members of the same program and from related ones. Stakeholders may be eventually involvedones. Stakeholders may be eventually involved

Annually based and the report contains advice Annually based and the report contains advice for improvement for improvement

M&E activities at M&E activities at different planning unitsdifferent planning units

Half-term evaluation:Half-term evaluation: toto strengthen strengthen the the project project resultsresults

Identifying achievements and detecting Identifying achievements and detecting delays in the project outcomesdelays in the project outcomes

Focusing on specific projects performance and Focusing on specific projects performance and their impacts on the strategy at higher their impacts on the strategy at higher organizational levels organizational levels

It will be done by INTA’s national directorate It will be done by INTA’s national directorate and involves the program advisory board and involves the program advisory board (stakeholders)(stakeholders)

3-year based process and the report contains 3-year based process and the report contains advice for improvement advice for improvement

M&E activities at M&E activities at different planning unitsdifferent planning units

Final evaluationFinal evaluation: to strengthen national programs : to strengthen national programs /regional centers strategy/regional centers strategy

Identifying the alignment of the planning units with Identifying the alignment of the planning units with the program/research center strategic statementsthe program/research center strategic statements

Bottom-up process at different levels of complexity Bottom-up process at different levels of complexity

6-year based process and comprises two stages:6-year based process and comprises two stages:

- - self-evaluationself-evaluation carried out by the management carried out by the management team and the advisory/regional board team and the advisory/regional board

- e- external (ex-post) evaluation xternal (ex-post) evaluation through peer through peer review review and includes the self evaluation report and includes the self evaluation report as an input as an input

Final Comments Final Comments

The institution faces a great opportunity on how The institution faces a great opportunity on how M&E should be done in this portfolio. A shift is M&E should be done in this portfolio. A shift is required to ensure that M&E becomes a culture required to ensure that M&E becomes a culture process in which development actors gain process in which development actors gain insights into how they can improve their insights into how they can improve their performance performance

The M&E system uses traditional tools but adds a The M&E system uses traditional tools but adds a new focus to deal with complex systems, the new focus to deal with complex systems, the territories and helping to achieve social territories and helping to achieve social objectives objectives

With a growing budget, INTA has strengthened With a growing budget, INTA has strengthened the base for a better control and effectiveness of the base for a better control and effectiveness of its actionsits actions

The ChallengeThe Challenge

We are building a M&E system to guide We are building a M&E system to guide the results towards the institutional the results towards the institutional objectives and develop an objectives and develop an organizational culture aligned with the organizational culture aligned with the territorial development strategyterritorial development strategy

This will enable us to contribute more This will enable us to contribute more effectively to achieve technological effectively to achieve technological innovation with social equityinnovation with social equity

Thank you for your attentionThank you for your attention

e-mails: e-mails: mirassou.susana@inta.gob.ar mirassou.susana@inta.gob.ar

bedascarrasbure.e@inta.gob.arbedascarrasbure.e@inta.gob.ar