cobp experimentation cover - nps · pdf file g. an experimentation venue is not an...

Click here to load reader

Post on 07-Jun-2020




0 download

Embed Size (px)


  • COBP Experimentation_Cover.PSD

  • About the CCRP

    The Command and Control Research Program (CCRP) has the mission of improving DoD’s understanding of the national security implications of the Information Age. Focusing upon improving both the state of the art and the state of the practice of command and control, the CCRP helps DoD take full advantage of the opportunities afforded by emerging technologies. The CCRP pursues a broad program of research and analysis in information superiority, information operations, command and control theory, and associated operational concepts that enable us to leverage shared awareness to improve the effectiveness and efficiency of assigned missions. An important aspect of the CCRP program is its ability to serve as a bridge between the operational, technical, analytical, and educational communities. The CCRP provides leadership for the command and control research community by:

    n articulating critical research issues; n working to strengthen command and control

    research infrastructure; n sponsoring a series of workshops and symposia; n serving as a clearing house for command and

    control related research funding; and n disseminating outreach initiatives that include the

    CCRP Publication Series.

  • This is a continuation in the series of publications produced by the Center for Advanced Concepts and Technology (ACT), which was created as a “skunk works” with funding provided by the CCRP under the auspices of the Assistant Secretary of Defense (C3I). This program has demonstrated the importance of having a research program focused on the national security implications of the Information Age. It develops the theoretical foundations to provide DoD with information superiority and highlights the importance of active outreach and dissemination initiatives designed to acquaint senior military personnel and civilians with these emerging issues. The CCRP Publication Series is a key element of this effort.

    Check our Web site for the latest CCRP activities and publications.

  • DoD Command and Control Research Program ASSISTANT SECRETARY OF DEFENSE (C3I)


    Mr. John P. Stenbit


    Dr. Linton Wells, II



    Dr. David S. Alberts

    Opinions, conclusions, and recommendations expressed or implied within are solely those of the authors. They do not necessarily represent the views of the Department of Defense, or any other U.S. Government agency. Cleared for public release; distribution unlimited.

    Portions of this publication may be quoted or reprinted without further permission, with credit to the DoD Command and Control Research Program, Washington, D.C. Courtesy copies of reviews would be appreciated.

    Library of Congress Cataloging-in-Publication Data

    Code of Best Practice for Experimentation / David S. Alberts. p. cm. Includes bibliographical references and index. ISBN 1-893723-07-0 (pbk.)



    Information Age Transformation Series

  • i

    Code of Best Practice for Experimentation

    Table of Contents

    Title Page No.

    Outline of Chapters ii

    List of Figures v

    Acknowledgements vi

    Preface viii

    Chapter 1: Introduction 1-1

    Chapter 2: Transformation and Experimentation 2-1

    Chapter 3: Overview of Experimentation 3-1

    Chapter 4: The Logic of Experimentation Campaigns 4-1

    Chapter 5: Anatomy of a Experiment 5-1

    Chapter 6: Experiment Formulation 6-1

    Chapter 7: Measures and Metrics 7-1

    Chapter 8: Scenarios 8-1

    Chapter 9: Data Analysis and Data Collection Plans 9-1

    Chapter 10: Conduct of the Experimentation 10-1

    Chapter 11: Products 11-1

    Chapter 12: Modeling-Based Experimentation 12-1

    Chapter 13: Adventures in Experimentation 13-1

    Appendix A: Measuring Performance and Effectiveness in the Cognitive



    Appendix B: Measurement Hierarchy B-1

    Appendix C: Modeling and Simulation C-1

    Appendix D: Situation Awareness Questionnaire D-1

  • ii


    Chapters Page No.

    1. Introduction 1-1 A. Why DoD Experiments? 1-1 B. Need for a Code of Best Practice 1-1 C. Scope and Focus 1-2 D. Organization of the Code 1-2

    2. Transformation and Experimentation 2-1 A. Transformation and NCW 2-1 B. NCW and Experimentation 2-1 C. Tenets of NCW 2-2 D. The Domains of NCW 2-2 E. Concept-Based, Mission Capability-Focused Experimentation 2-5

    3. Overview of Experimentation 3-1 A. Uses of Experiments 3-1 B. Experimentation Campaigns 3-4 C. Discovery Experiments in the Context of an Experimentation

    Campaign 3-7

    D. Hypothesis Testing Experiments in the Context of an Experimentation Campaign


    E. Demonstration Experiments in the Context of an Experimentation Campaign


    F. Results of a Well-Crafted Experimentation Campaign 3-13 G. An Experimentation Venue Is Not An Experimentation

    Campaign 3-14

    H. Core Challenges of Experimentation 3-14

    4. The Logic of Experimentation Campaigns 4-1 A. Why Campaigns Rather Than Individual Experiments? 4-1 B. Experimentation Campaigns Require a Different Mindset 4-2 C. Structure Underlying Experimentation Campaigns 4-6 D. Maturity of Knowledge Contribution 4-6 E. Fidelity of Experimentation Settings 4-9 F. Complexity of Issues Addressed 4-12 G. Designing Experimentation Campaigns 4-13 H. Conclusion 4-14

    5. Anatomy of an Experiment 5-1 A. Phases of an Experiment 5-1 B. Pre-Experiment Phase 5-3 C. Formulation of the Experiment 5-4

  • iii

    D. Establishing the Experimentation Team 5-9 E. Detailed Experiment Plan 5-26 F. Conduct of the Experiment 5-29 G. Experimentation Environment Setup 5-29 H. Pre-test or Rehearsal 5-30 I. Execution of the Experiment 5-32 J. Post-Experiment Phase 5-36 K. Analysis 5-37 L. Integrating Experiment Results 5-39 M. Interpretation of the Results 5-39 N. Archiving and Circulating Data and Products 5-41 O. Modeling and Simulation in Support of Experiments 5-42

    6. Key Experimentation Design Considerations 6-1 A. Focus 6-1 B. Propositions and Hypotheses 6-1 C. Making Better Sets of Propositions and Hypotheses 6-2 D. Propositions for an Exploratory Experiment in Self-

    Synchronization 6-3

    E. Initial Models and Their Use 6-5 F. Initial Executable Model 6-7 G. Executable Model for the Self-Synchronization Experiment 6-8 H. Ideal Versus Real Transformational Experimentation 6-9 I. Ideal Experimentation 6-10 J. Principles of Experimentation 6-11 K. Transformational Experimentation Realities 6-17

    7. Measures and Metrics 7-1 A. Focus 7-1 B. Definitions 7-1 C. Criteria for Measures and Metrics 7-5 D. Process of Measure and Metric Selection 7-8 E. Established Measures and Metric Systems 7-12 F. Example Application: Experiment in Self-Synchronization 7-17 G. Conclusion 7-29

    8. Scenarios 8-1 A. What is a Scenario? 8-1 B. Scenarios in Transformation Experimentation 8-1 C. Scenario Selection, Adaptation, and Creation 8-9 D. Example Application: Self-Synchronization Experiment 8-13 E. Conclusion 8-15

    9. Data Analysis and Collection Plans 9-1 A. Purpose and Focus 9-1 B. Relationship Between the Data Analysis Plan and the Data 9-1

  • iv

    Collection Plan C. Data Analysis Plans 9-2 D. Data Collection Plans 9-13

    10. Conduct of the Experiment 10-1 A. Purpose 10-1 B. Pre-test 10-1 C. Pre-experiment Data Collection 10-4 D. Training 10-5 E. Working with Observers and Controllers 10-9 F. Data Collection 10-11 G. Data Capture and Archiving 10-14 H. In-Process and Quick Look Reporting 10-16 I. Example Experiment: Self-Synchronization 10-17

    11. Products 11-1 A. Types of Documentation Products and Their Use 11-1 B. Archiving the Experiment for Future Investigations 11-10 C. Key Considerations: Products and Their Dissemination 11-14 D. Tying it all Together: Documenting Experimentation Campaigns 11-22 E. Conclusion 11-23

    12. Model-Based Experiments 12-1 A. Overview 12-1 B. The Anatomy of a Model-Based Experiment 12-5 C. Pre-Experiment Phase 12-5 D. Executable Models (Simulations) 12-10 E. Conduct of the Experiment 12-11 F. Post-Experiment Phase 12-13 G. Analysis 12-13 H. Counterintuitive Model Results 12-15 I. Issues in Model-Based Experimentation 12-16 J. Conclusion 12-19

    13. Adventures in Experimentation: Common Problems and Potential Disasters


    A. Purpose 13-1 B. Flawed Experimentation Environment 13-2 C. Lack of Adequate Resources 13-4 D. Flawed Formulation 13-5 E. Flawed Project Plan 13-8 F. Measurement/Analysis Problems 13-11 G. Post-Experiment Problems 13-14 H. Issues in Campaigns 13-15 I. Conclusions 13-17

  • v

    List of Tables & Figures

    Number Caption Page No.

    Fig 2-1 NCW Value Chain 2-3

    Fig 2-2 Mission Capability Packages 2-6

    Fig 2-3 Maturity Model 2-7

    Fig 3-1 From Theory to Practice 3-6

    Table 4-1 Comparison of An Experiment Versus An Experimental Campaign


    Fig 4-1 The Experimentation Campaign Space 4-8

    Fig 5-1 Phases of Experiments 5-2

    Fig 5-2 Modeling and Simulations 5-45

    Fig 6-1 Illustrative Initial Descriptive Model: Self-Synchronization Experiment


    Fig 8-1 Scenario Threat Space 8-3

    Fig 11-1 Management and Dissemination