Multi-organisational collaboration in operations assessment and

evaluation: Prospects, challenges, and implications for practice

29th ISMOR, 2012

Andy Williams Operational Analysis Branch

HQ Supreme Allied Commander Transformation

1

The views expressed in this article are the views of the author and do not necessarily represent the views of

NATO or any other organisation

2

ACT is NATO’s leading agent for change, driving, facilitating, and advocating continuous improvement of Alliance capabilities to maintain and enhance the military relevance and effectiveness of the Alliance.

Allied Command Transformation

3

ACT Trident

Strategic

Thinking

Capability

Development

Education &

Training

Interoperability

Coherence

Partners

+

Outreach

Lessons

Learned

Lessons

Identified

Abstract

Measuring progress and results of an organisation’s activities—“assessment” in the military and “evaluation” in the civilian sector—are typically arranged as principal-agent relationships between a management, leadership, funding or accountability body, and the evaluating body. Recent practice has seen the emergence of collaboration between assessment and evaluation organisations involving more complex arrangements with either multiple principals or multiple agents, or both. Furthermore, NATO has expressed an ambition for military operations assessment organisations to work more closely with civilian evaluation agencies in conflict or fragile environments.

This paper proposes a framework in which military assessment organisations can interact with civilian evaluation departments. The framework considers: the rationale for collaborating on operations assessments; the necessary pre-conditions; the challenges and transaction costs involved; and the collaboration process, including structural issues of governance and administration, mechanisms of interaction, and methodology. Ideas for empirical research and further theoretical development are presented.

Keywords: Operations Assessment, Monitoring and Evaluation, Evaluation, Collaboration

4

Operations Assessment

Operations Assessment

“the function that enables the measurement of progress and results of operations in a military context, and the subsequent development of conclusions and recommendations that support decision making” (COPD, p. 5-1).

A.k.a. campaign assessment; operational assessment, assessment

Many similarities to “monitoring and evaluation”

Assessment Capability Development Plan

Concept development

Refinement of NATO policy and directives

Experimentation

Development of new training

Analytical support

Capture of current problems in assessment

Lessons identified / learned

Research projects

Literature reviews, workshops, interviews

6

Context

Problems in operations assessment

Rationale for increased collaboration

Multi-organisational collaboration framework

Conclusions and future work

7

Agenda

8

Current challenges in Ops Assessment

Data limitations Evaluating causal impacts Methodological stovepiping Drawdowns and transition

Abundance of security-incident related data Allows granular measurement of change Directly relatable to plan elements

Abundance of other military data and reports Allow granular measurement Not easily relatable to plan elements

Abundance of governance, development, economic, social, regional data Allows retrospective, strategic level assessments Not directly relatable to plan elements

Conclusion: militaries are planning for governance and development impacts, but have limited data to assess progress

9

Data limitations

10

Evaluating causal impacts

Current military plans clearly call for evaluation of complex causalities, which isn’t possible without access to relevant non-military data

Military missions eventually drawdown

Political requirements for assessments do not

Transition period is especially challenging for data accessibility

11

Drawdowns

“Logframe” approaches dominate

Assumptions

Relatively fixed objectives

Statistical reasoning

12

Methodological stovepipes

13

Increasing collaboration in operations assessment

Pre-conditions Process Structures Mechanisms Challenges

What are the relevant organisations? National aid and development agencies

Independent evaluation units

International organisations

NGO

Private companies

…

Focus initially on national government organisations Development agencies

Evaluation agencies

14

Collaborative partners?

M & E (OECD DAC Definition)

Evaluation: The systematic and objective assessment of an on-going or completed project, programme or policy, its design, implementation and results.

Monitoring: A continuing function that uses systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progress and achievement of objectives andprogress in the use of allocated funds.

15

16

Collaborative Framework

Ansell, C.& Gash, A. (2007). Collaborative governance in theory and in practice. Journal of Public

Administration Research and Theory, 18, 543-571.

Partners in Cooperation

Understand who should be cooperating?

Spectrum of Interaction

Understand that cooperation is a spectrum

Organisational level of interaction

From tactical to strategic?

Circumstances of Partnerships

Out of mission or in mission?

Can real cooperation be “engineered?”….

17

Key Principles

Understand the Type of Evaluation

Develop Networks

Formalise Agreements

Designing Cooperative Assessments

Mission Coordination

Burden sharing

Consider Other Assessment Processes

18

Mechanisms

Process

19

Tactical HQs

JF HQ

SHAPE HQ

NATO HQ

ASSESSREPs

Campaign MOE

Strategic MOE

ASSESSREPs

Campaign MOE

Strategic MOE

PMR

Organisations

Pol-Mil Review Sprt to Plans & Ops

Products Focus

End-state and political objectives

Strategic Military Objectives / Effects

Operational Military Objectives / Effects

Tactical objectives / mission / tasks

Data exchanges

Survey questions and other data collection instruments

Data sources

Final Reports

Staff

Lessons learned, best practises, methods, and terminology

20

Content of Collaboration

21

Conclusions and Way Ahead

Cooperative Assessment Benefits

Operational Benefits: Data exchange and multiple perspectives Access to expertise Reducing duplication of resources Improving quality and credibility of results

Political benefits: Adherence to international standards Political will to demonstrate effectiveness of interventions Coherence in international community

Improving Cooperation: Increasing interaction and dialogue outside of missions

Organisational Learning: Exchange of techniques and best practices

22

Challenges

23

“Transaction” costs of collaboration Extra resources Legal / administrative factors Travel Networking

Analysis process challenges Agreeing on scope, objectives and format Methodology Team compositions Timelines

Political sensitivities Controversial findings Reduction in sensitivity

Campaign assessment = program evaluation

Ops researchers / analysts as social scientists?

Development of models, theories and frameworks + research

Cooperative assessment changes nature of analysis activity

24

Thoughts…

Development of NATO concept

Cooperative Assessment with External Actors

October 2012 release to NATO Nations

Next version of the NATO Operations Assessment Handbook

Update to the NATO Comprehensive Operations Planning Directive

Pilot Operations Assessment training course

Latvian Defence College, Riga, 12 – 16 Nov 2012

25

Future work

26