practical profiles for managing systems engineering r&d

11
IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 56, NO. 2, MAY2009 341 Practical Profiles for Managing Systems Engineering R&D William W. Agresti, Senior Member, IEEE, and Richard M. Harris, Senior Member, IEEE Abstract—A new and practical way of profiling R&D projects is presented. The operational context is the investment of discre- tionary R&D funds by systems engineering firms. In this work program positioning profile, R&D projects are identified as one of four types: targeting, reinforcing, enabling, or remodeling, accord- ing to the relationship between the R&D project and the systems engineering work program. A portfolio map is introduced to pro- vide a visual representation of a set of R&D projects and to show likely transitions between project types over time. The new profile and portfolio map can support management efforts to character- ize and assess an R&D portfolio for its contribution to ongoing systems engineering work. In addition to this new project pro- file, other profiling strategies are discussed in the framework of an R&D balanced scorecard. The multiple profiles are a natural consequence of the inherent performance multidimensionality of the R&D activity in the wide-ranging discipline and practice of systems engineering. Index Terms—Balanced scorecard (BSC), R&D management, R&D project portfolio, R&D projects. I. INTRODUCTION T HIS PAPER introduces a new and practical way of profil- ing R&D projects in systems engineering. This new pro- file evolved over several years of experience by the authors and colleagues who were responsible for evaluating proposals for R&D investments, making recommendations and decisions on projects to fund, reviewing the progress of ongoing R&D projects, and briefing executives on the nature, content, struc- ture, and potential benefits of the R&D portfolio. This last ac- tivity, attempting to characterize collections of R&D projects along different dimensions, was especially a motivator to create improved profiling techniques. Projects were being discussed in terms of dimensions such as risk and expected reward, but there was a need to be able to discuss the set of projects as they related to each other and to the mission of the organization. Fur- thermore, there was a need to create concise representations for senior management to help explain how the investments in R&D projects may be seen as supporting the current work program and expanding it in the future. This new profile emerged, in a trial-and-error process, from years of professional practice as a management aid that was very useful for characterization and evaluation of the R&D portfolio. Manuscript received March 6, 2007; revised March 4, 2008, June 2, 2008, and July 18, 2008. First published March 21, 2009; current version published April 17, 2009. Review of this manuscript was arranged by Department Editor J. K. Pinto. W. W. Agresti is with Carey Business School, Johns Hopkins University, Baltimore, MD 21201 USA (e-mail: [email protected]). R. M. Harris, retired, Huntingtown, MD 20639 USA (e-mail: booknet@ rmharris.com). Digital Object Identifier 10.1109/TEM.2009.2013825 The specific notion of a profile and the motivation for it should be made clear at the outset. By a profile is meant a representation intended to highlight certain characteristics of the entities being examined. Profiles have been shown to be useful in representing portfolios of R&D projects because the collection of projects and the entire R&D program needs to be understood from many different perspectives and for different audiences and stakehold- ers. For example, an obvious profile of interest would represent the projects according to their relationship to marketable prod- ucts and services. Certain projects may be pursuing technologies that could lead to products within, say, a two- to three-year time frame. Other projects may be investigating foundational science that underlies an organization’s products, so these projects re- flect the pursuit of basic science. The path from such projects to actual products is much longer. A corresponding profile then would highlight this dimension, the basic-to-applied nature of the projects, to show the relative number of projects and amount of R&D funds plotted along this dimension. Such a portrayal will facilitate understanding, for example, if the portfolio ap- pears to be more strongly invested in long-term R&D than was the firm’s objective. If profiles of R&D portfolios are in common use, why is another one being proposed? There are three reasons. The profile introduced here: 1) is oriented to systems engineering and to organizations that provide related products and services; 2) highlights the relationship of the R&D projects to the ongoing professional work program; 3) has been demonstrated to be useful in the practice of man- aging R&D programs. Characterization of R&D portfolios in this context—oriented to the systems engineering domain and highlighting the rela- tionship to the work program—poses specific challenges that are discussed in this paper. A new profile addressing these challenges may be a useful contribution to the R&D and sys- tems engineering management communities. In the process of introducing the new profile, this paper will review and orga- nize other approaches for characterizing collections of R&D projects. The remainder of this paper is organized as follows. This section concludes with a review of relevant literature and prac- tice and a discussion of the systems engineering R&D envi- ronment. Section II introduces the new work program posi- tioning profile and a visualization of R&D projects, called a portfolio map. Section III reviews other R&D project profil- ing approaches and uses a balanced scorecard (BSC) to or- ganize them for an R&D setting. Section IV discusses the conclusions. 0018-9391/$25.00 © 2009 IEEE

Upload: rm

Post on 24-Sep-2016

213 views

Category:

Documents


1 download

TRANSCRIPT

IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 56, NO. 2, MAY 2009 341

Practical Profiles for Managing SystemsEngineering R&D

William W. Agresti, Senior Member, IEEE, and Richard M. Harris, Senior Member, IEEE

Abstract—A new and practical way of profiling R&D projectsis presented. The operational context is the investment of discre-tionary R&D funds by systems engineering firms. In this workprogram positioning profile, R&D projects are identified as one offour types: targeting, reinforcing, enabling, or remodeling, accord-ing to the relationship between the R&D project and the systemsengineering work program. A portfolio map is introduced to pro-vide a visual representation of a set of R&D projects and to showlikely transitions between project types over time. The new profileand portfolio map can support management efforts to character-ize and assess an R&D portfolio for its contribution to ongoingsystems engineering work. In addition to this new project pro-file, other profiling strategies are discussed in the framework ofan R&D balanced scorecard. The multiple profiles are a naturalconsequence of the inherent performance multidimensionality ofthe R&D activity in the wide-ranging discipline and practice ofsystems engineering.

Index Terms—Balanced scorecard (BSC), R&D management,R&D project portfolio, R&D projects.

I. INTRODUCTION

THIS PAPER introduces a new and practical way of profil-ing R&D projects in systems engineering. This new pro-

file evolved over several years of experience by the authorsand colleagues who were responsible for evaluating proposalsfor R&D investments, making recommendations and decisionson projects to fund, reviewing the progress of ongoing R&Dprojects, and briefing executives on the nature, content, struc-ture, and potential benefits of the R&D portfolio. This last ac-tivity, attempting to characterize collections of R&D projectsalong different dimensions, was especially a motivator to createimproved profiling techniques. Projects were being discussedin terms of dimensions such as risk and expected reward, butthere was a need to be able to discuss the set of projects as theyrelated to each other and to the mission of the organization. Fur-thermore, there was a need to create concise representations forsenior management to help explain how the investments in R&Dprojects may be seen as supporting the current work programand expanding it in the future. This new profile emerged, in atrial-and-error process, from years of professional practice as amanagement aid that was very useful for characterization andevaluation of the R&D portfolio.

Manuscript received March 6, 2007; revised March 4, 2008, June 2, 2008,and July 18, 2008. First published March 21, 2009; current version publishedApril 17, 2009. Review of this manuscript was arranged by Department EditorJ. K. Pinto.

W. W. Agresti is with Carey Business School, Johns Hopkins University,Baltimore, MD 21201 USA (e-mail: [email protected]).

R. M. Harris, retired, Huntingtown, MD 20639 USA (e-mail: [email protected]).

Digital Object Identifier 10.1109/TEM.2009.2013825

The specific notion of a profile and the motivation for it shouldbe made clear at the outset. By a profile is meant a representationintended to highlight certain characteristics of the entities beingexamined. Profiles have been shown to be useful in representingportfolios of R&D projects because the collection of projectsand the entire R&D program needs to be understood from manydifferent perspectives and for different audiences and stakehold-ers. For example, an obvious profile of interest would representthe projects according to their relationship to marketable prod-ucts and services. Certain projects may be pursuing technologiesthat could lead to products within, say, a two- to three-year timeframe. Other projects may be investigating foundational sciencethat underlies an organization’s products, so these projects re-flect the pursuit of basic science. The path from such projectsto actual products is much longer. A corresponding profile thenwould highlight this dimension, the basic-to-applied nature ofthe projects, to show the relative number of projects and amountof R&D funds plotted along this dimension. Such a portrayalwill facilitate understanding, for example, if the portfolio ap-pears to be more strongly invested in long-term R&D than wasthe firm’s objective.

If profiles of R&D portfolios are in common use, why isanother one being proposed? There are three reasons. The profileintroduced here:

1) is oriented to systems engineering and to organizationsthat provide related products and services;

2) highlights the relationship of the R&D projects to theongoing professional work program;

3) has been demonstrated to be useful in the practice of man-aging R&D programs.

Characterization of R&D portfolios in this context—orientedto the systems engineering domain and highlighting the rela-tionship to the work program—poses specific challenges thatare discussed in this paper. A new profile addressing thesechallenges may be a useful contribution to the R&D and sys-tems engineering management communities. In the process ofintroducing the new profile, this paper will review and orga-nize other approaches for characterizing collections of R&Dprojects.

The remainder of this paper is organized as follows. Thissection concludes with a review of relevant literature and prac-tice and a discussion of the systems engineering R&D envi-ronment. Section II introduces the new work program posi-tioning profile and a visualization of R&D projects, called aportfolio map. Section III reviews other R&D project profil-ing approaches and uses a balanced scorecard (BSC) to or-ganize them for an R&D setting. Section IV discusses theconclusions.

0018-9391/$25.00 © 2009 IEEE

342 IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 56, NO. 2, MAY 2009

A. Review of the Literature and Practice

An abundant literature exists in R&D project selection andportfolio structuring [1]–[4]. Often the focus is on developingand refining decision models that can be quite sophisticated,such as those based on real options [5], game theory [6], the-ory of constraints [4], and mathematical programming [7], [8].The purpose in citing the use of these approaches to decisionmaking is not to discuss them here; references are provided tosupport further inquiry. Instead, the purpose is to highlight thestriking difference between the sophistication of such publishedmethods and the relative simplicity of methods that are actuallyused in professional practice. Schmidt and Freeland note that, inthe literature, “Project selection has traditionally been formal-ized as a constrained optimization problem,” but that “ClassicalR&D project-selection models have been virtually ignored byindustry” [9]. As an indication that very simple methods are theones being used in practice, a survey by Cooper et al. found that40% of businesses use bubble diagrams, in which the projectsare represented as circles (bubbles) and simply plotted on a 2-Dgrid, based on dimensions such as risk and reward [1].

More specifically relevant to the current research are the manypublished reports on strategies for profiling, structuring, andcharacterizing portfolios of R&D projects. Both the researchliterature and professional practice reveal many different profilesbeing proposed and used. The abundance of profiles is due to themultidimensionality of the R&D activity. As Osama observes,“the problem of performance multidimensionality . . . is mostsevere in research and development (R&D) settings due to theinherent multi-dimensionality of R&Ds output and the long-term and intangible nature of the process itself” [10], p. vii].The notion here of multidimensionality is simply that the R&Dactivity cannot be evaluated by a single performance metric.And because there are multiple dimensions and metrics of theR&D performance, it makes sense that it might be helpful to usemultiple profiles to characterize the various dimensions.

For profiling R&D projects, financial measures are the mostwidely used methods, by 77.3% of businesses [1]. The R&Dproject’s strategic fit with corporate priorities is also a verypopular profiling dimension, with 64.8% of companies usingthis approach [1]. In addition to profiles having cross-industryapplicability, profiles that are industry-specific may be particu-larly relevant. For example, in the pharmaceutical and biotechindustries, a meaningful profile of R&D projects is a 2-D mapshowing how each project is rated on efficacy versus safety [11].Within an industry, individual organizations establish profilesthat are important to them. For example, the research divisionof PEMEX, Mexico’s oil and gas company, tracks 26 technol-ogy areas. It has identified four of the areas as “core” technologyarenas, with the remaining classified as maintenance arenas. Itprofiles its projects as being core or maintenance because it hasa policy to allocate 75% of its resources to core arenas [12].

B. R&D in Systems Engineering Firms

An immediate challenge for effective R&D in systems engi-neering is the breadth of the discipline and associated profes-

sional practice. Definitions from the literature and the leadingprofessional society in the field illustrate this breadth, definingsystems engineering as:

1) “. . . the design, production, and maintenance of trustwor-thy systems within cost and time constraints” [13, p. 10];

2) “an interdisciplinary approach and means to enable therealization of successful systems” [14].

R&D projects in support of systems engineering can reflectthe breadth indicated in these definitions. Within the technologyspace alone, projects can include the entire range of comput-ing, telecommunications, sensor, and information technologyhardware, software, and systems, as well as technologies asso-ciated with application domains of interest to the firm and itscustomers. Attempts to have the R&D program address the un-derlying sciences and potential innovations across such a widespectrum can lead to a portfolio of too many projects that arepoorly funded.

So as not to focus on the R&D at individual companies,consider the research agenda of the International Council onSystems Engineering (INCOSE) [15]. The following are exam-ples of topics from the INCOSE research agenda, selected toconvey the wide range of R&D possibilities [15].

1) “Develop models of the production, integration, andflow of information within the system developmentprocess.”

2) “Search for a grand unification theory for the developmentof man-made systems.”

3) “Compare and contrast allocation methods and techniquesfor allocating functions to components in design basedupon performance and cost requirements.”

4) “Develop interface management tools that includerequirements allocation, and performance and costtrades.”

5) “Develop formal methods for stating requirements and aformal proof-of-correctness method for requirements.”

6) “Create a mathematical representation of a system’s in-puts, outputs, functions, and components for use in systemspecification and design.”

7) “Develop a robust application method of decision theoryfor system trade-offs.”

8) “Develop a tool that implements the integrated risk anal-ysis method for product architecture development.”

9) “Examine and contrast theories of human systems andhow they create products.”

10) “Conduct empirical research of alternate methods for in-stilling a shared vision into a team.”

11) “Develop a standard interface that allows SE tools to in-tegrate with legacy and COTS products.”

Systems engineering, as used in this paper, reflects thisbreadth of considerations, which, in turn, further motivatesthe need for organizations to be able to characterize theirR&D projects in multiple ways in an attempt to match thisbreadth. For example, one profiling dimension may show thedistribution of R&D funds across the technology, human,management, quality, cost, and process aspects of systemsengineering.

AGRESTI AND HARRIS: PRACTICAL PROFILES FOR MANAGING SYSTEMS ENGINEERING R&D 343

In addition to the breadth issue, another R&D challenge insystems engineering firms, when contrasted with R&D in otherorganizations, is that the funds available for the entirety ofdiscretionary investments may be very modest. Firms that donot have margins generated by product lines and instead relysolely on professional services typically have limited funds forR&D. For example, using 2005 data in the IEEE Spectrum R&D100 [16], the top ten companies in R&D expenditures as a per-centage of sales averaged 18.9%, compared to 4.01% for the topten systems integrators.1

Much of the R&D literature is oriented to operational con-texts different from that of systems engineering. For example,manufacturing companies may rely on their R&D for process in-novations to become more cost-competitive. In pharmaceuticalcompanies, R&D is counted on as the source of new productsand is routinely positioned at the earliest stage of a typicallylengthy product development life cycle. The benefits of the R&Dinvestments are in the value created downstream.

With systems engineering, the value chain to revenue is notgenerally so clear. While the outcomes of systems engineeringR&D projects may indeed translate into new products or ser-vices, the nature of the projects may mean that the results are dif-fused as innovative techniques or capabilities throughout a workprogram and across a base of customers. The mission of the firmis to deliver the highest quality systems engineering products,systems, and services. The extent to which customers want orexpect highly innovative, state-of-the-art solutions varies widelybecause customers vary and the nature of the systems they needvary as well. Some customers want systems that use more ma-ture and less risky technology. Key to understanding the role ofinnovation is the risk appetite of the customer, the technical andmanagement challenges inherent in the planned systems, theperceived benefit of being positioned as a leading-edge technol-ogy provider, and the comfort level relying upon immature butpromising technologies.

The preceding discussion sets the context for viewing thispaper as different for its focus on systems engineering R&Dand its bias for practical approaches. It is proposed that therelationship of the R&D program to the work program of a sys-tems engineering firm is worth exploring, with the idea that awell-conceived and managed R&D program may be critical tothe success of the firm. In particular, this paper offers a newand practical way of profiling R&D projects in systems engi-neering firms, highlighting the positioning of the R&D projectsrelative to the systems engineering work program. While the ob-servations here are intended to be applicable to R&D programsgenerally, the specific experience base that led to this profilingmethod is the authors’ management of in-house R&D, some-times referred to as internal R&D (or IRAD) programs at severalorganizations. In these R&D programs, decisions are made toinvest discretionary funds in projects that are most often man-aged and executed by internal investigators and teams from thefirm. With this practical orientation, the profile proposed hereis in the spirit of the contribution by Bordley [17] that offered

1Data from Federal Computer Week online, “Top 74 Systems Integrators,”September 4, 2006 issue, downloaded from http://www.fcw.com/article95875–09-04–06-Print, March 5, 2007, using the top ten public companies for whichdata were available.

exceedingly practical, incisive, and revealing questions aboutR&D projects as a way to help generate higher quality R&Dprojects in an organization.

II. PROFILE AND PORTFOLIO MAP OF SYSTEMS ENGINEERING

R&D PROJECTS

This section introduces the new profile and a way of visu-alizing the projects that are profiled. Experiences with usingthe profile and recommendations for its use in the future arediscussed.

A. Work Program Positioning Profile

This profile of systems engineering R&D projects is designedto highlight the relationship of the project to the firm’s systemsengineering practices and ongoing work program. Each projectis identified as being in one of four categories summarized nextand in Table I.

1) Targeting (get-a-job)—The firm may not be perceivedby customers as among the set of performing organiza-tions in a given work area. While it may have the rel-evant knowledge and skill base, the firm needs an inte-grating project to demonstrate that it can be an effectiveperforming organization competing for work in a speci-fied area. For example, the firm may have skilled profes-sionals in system quality assurance and testing, but thosepractices are being applied as part of development andintegration efforts for customers. A targeted R&D projectmay coalesce into a distinctive and novel IndependentVerification and Validation (IV&V) capability that buildson the development and integration experiences. The re-sulting integrated methodology may be the basis for bid-ding on IV&V contracts and getting a job in the targetedarea.

2) Reinforcing (keep-a-job)—These are R&D projects in-tended to reinforce the current working relationship witha customer or set of customers. The motivation for suchprojects may be that there is a concern about the customer’ssatisfaction with the delivered systems engineering sup-port. By investing in such projects, the firm can say thatthe R&D project is evidence that the customer is so impor-tant to the firm that it has invested its discretionary fundsin ways that are consistent with the customer’s mission.The investment will further enhance the expertise of thefirm’s professional staff in ways that develop the “benchstrength” of the firm in subjects relevant to the customer.Reinforcing projects can easily strain the definition ofR&D to the point where they are sometimes colloquiallyreferred to as R&D with a “lower case” R, i.e., with littleexpectation that the research rises to the level of publish-able or state of the art. An obvious indicator of successwith reinforcing projects is whether, in fact, the firm doesretain the work with the customer. An even better outcomeis if reinforcing projects lead to an expanded work pro-gram. Success on these projects has the effect of “turningaround” or growing a work program that was perceived asvulnerable to being reduced or cancelled.

344 IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 56, NO. 2, MAY 2009

TABLE IR&D PROJECT TYPES BASED ON RELATIONSHIP TO SYSTEMS ENGINEERING WORK PROGRAM

3) Enabling (do-a-better-job)—These R&D projects gener-ate results or deliverables that create or enhance enablingcapabilities that will span a significant portion of systemsengineering practice areas. R&D projects of this type are

often tools, methodologies, or systems that can be appliedbroadly across the work program. For example, systemsintegrators are repeatedly faced with challenges in qual-ity assurance, configuration management, and testing. An

AGRESTI AND HARRIS: PRACTICAL PROFILES FOR MANAGING SYSTEMS ENGINEERING R&D 345

TABLE I (continued)R&D PROJECT TYPES BASED ON RELATIONSHIP TO SYSTEMS ENGINEERING WORK PROGRAM

R&D project might develop a suite of innovative and flexi-ble test harnesses that could be adapted for use on multipleprojects for enhanced systems testing.

4) Remodeling (get-smart)—The firm’s mission and visionmay call for a knowledge base that does not exist in theorganization. A targeted R&D project may be defined to“get-smart” or retool in this new knowledge area. Tacti-cal questions include how best to “spin up” a capability:for example, to “grow it” through education of currentstaff, “buy it” through hiring staff with needed expertise,or “rent it” from consultants, university researchers, orother experts to collaborate with and mentor in-house pro-fessional talent during the capacity-building process. Re-modeling should be interpreted broadly to include devel-oping technology-intensive new business models. Whilethe success of an R&D project is traditionally seen as anew product, CEOs now see products as too susceptible tobeing copied by others at lower cost [18]. Instead, R&Dleaders should look to remodeling projects for craftinginnovative and transformative business models that havemore potential for sustainable strategic advantage.

The intention is that profiling R&D projects according tothese four categories will help an organization understand therelationship of the projects to the ongoing systems engineeringwork program. In addition to the examples mentioned before,the demonstrated benefit of the profile has been to clarify pre-cisely what relationships between R&D projects and systems

engineering work are most important. In its four categories ofR&D projects, the profile gives the result of that consideration,namely it has been most valuable to recognize whether the R&Dproject is intended to lead to the creation of new systems en-gineering work (targeting), to strengthen the likelihood that thefirm will sustain its current work program with the client (rein-forcing), to enhance the firm’s capabilities more broadly acrossmultiple clients and work programs (enabling), and to repositionthe firm to compete in the future (remodeling).

B. Portfolio Map for the Work Program Positioning Profile

Given the typical breadth of work programs in systems engi-neering firms, their R&D projects may not neatly fit a single oneof the four types mentioned before. It may be that displaying thework program positioning portfolio space will be informative toshow the number of projects of each type and the ways they areperceived as having characteristics of one or more types. Fig. 1visualizes sample project numbers (P10–P19) corresponding toR&D projects that are arrayed in the 2-D space according totype. For example, consider the position of project P19, mid-way between a remodeling and a targeting project. P19 maybe a demonstration project to remodel the organization (get-smart) in social networking technology and develop a showcaselaboratory that can be used to target new business (get-a-job),showing customers how popular social networking technologycan be brought inside customers’ organizations to facilitate theirknowledge sharing.

346 IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 56, NO. 2, MAY 2009

Fig. 1. Portfolio map of projects P10–P19 and transitions T1–T4.

A portfolio map like Fig. 1 can motivate useful discussionsabout the distribution of R&D projects by type. One key indica-tor of the health of the portfolio is the split between long-term(enabling and remodeling) and short-term (targeting and rein-forcing) projects. A portfolio dominated by short-term projectsshould be a warning sign that there may be too little atten-tion being paid to longer term technology trends. Too muchreinforcing may be symptomatic of fundamental work programdeficiencies such as shortfalls in staff or management capabil-ities. The R&D program may be captive to shoring up currentwork programs that are employing obsolete operational or tech-nological paradigms such as application development jobs forwhich the firm has missed transitions to service-oriented archi-tectures and Web services. An R&D program with a short-termfocus may be monopolizing funds that may be better invested incritical remodeling projects to keep the firm on the cutting edgeof changing technologies.

With annual budget cycles that are typical in systems engi-neering firms, R&D managers face the challenge of allocatingfunds between ongoing projects and new ones. Naturally, someR&D efforts require multiyear funding. As a result, the manageris faced with evaluating the progress of these current projects(do they deserve continued funding?) in light of proposals fornew projects. It is obvious that, in general, there are advantageswhen an R&D program can both continue the support of promis-ing ongoing projects and also launch new starts in response toexciting new proposals.

This consideration of funding both continuing and new R&Dprojects raises the question of examining project evolution overfunding time periods. For example, in the systems engineeringcontext, a project may have been funded in its first year asa remodeling project for the organization to “get-smart” in agiven technology or application domain. If it merits a secondyear of funding, is the project still just making the organizationsmarter or should it now be ready to prepare the organizationto target work and “get-a-job”? So the placement of a given

project in the portfolio map may be expected to change overtime. Conversely, a project that does not move (e.g., three yearsof “getting-smart”) may be an indicator of a lack of progressif the objective is to position the company to gain work in atechnology area.

Maintaining the portfolio map over time can help capture thetransitions of the R&D efforts. Fig. 1 shows arrows to suggestthe following transitions of focus and funds that may be expectedfrom one year to the next.

T1. Remodeling ⇒ Targeting: How much investment is neededbefore the firm can target its first work program that buildsupon the R&D results? After repeated investments in remod-eling projects, it may be time for a targeting R&D projectto prototype the business model or to showcase the technol-ogy in a form that can suggest how it may provide value forcustomers

T2. Targeting ⇒ Reinforcing: After a targeting R&D projectsucceeds in helping to win a work program, what reinforcingprojects are needed to sustain and grow it? It may be criticalto keep investigating technologies that are in rapid flux, forexample, to monitor the experiences of early adopters and totrack technology maturity from commercial vendors.

T3. Reinforcing ⇒ Enabling: Are work-program-specific rein-forcing R&D projects symptomatic of a need for cross-cuttingenabling projects that will build core capabilities with broaderimpact? Creative insight may identify an integrated projectthat will have a key by-product of enhancing interoperabilityacross technologies, such as systems specification languagetools that can also support eventual acceptance testing of asystem.

T4. Enabling ⇒ Remodeling: Are enabling R&D projects anindication of too much investment in enhancing legacy plat-forms and processes when a better approach may be to breakfree and launch remodeling projects to get into newer coreinfrastructures for the years ahead? Managers with visionmay solicit remodeling R&Ds to anticipate emerging tech-nologies, guided by an awareness of global trends, changingmarket forces, customers’ plans, and corporate vision.

The motivation for proposing the portfolio map in Fig. 1and discussing the potential transitions of projects over timeis that these considerations have been helpful to the authors.Managers may be more likely to see the relative balance inthe collection of projects among the four project types and theextent to which that balance makes sense in support of the entireorganization.

The work program positioning profile and the portfolio mapcan support management decision making in several phases ofthe R&D life cycle. In the planning phase, the profile and mapcan contribute to deciding on the statement of objectives of theR&D program each year: what is the desired distribution ofprojects (and R&D funds) among the four categories in the pro-file? For example, if there is a solicitation for R&D ideas andproposals, the solicitation statement released in one year mayexpress a preference for targeting proposals because the profilehas revealed that there is a gap in such projects compared to thefirm’s desire for them. During the life cycle phases involving the

AGRESTI AND HARRIS: PRACTICAL PROFILES FOR MANAGING SYSTEMS ENGINEERING R&D 347

execution of the R&D projects, the profile can serve the man-agement function of control by showing whether the projects aremaking progress in ways that are consistent with their identifi-cation as targeting, enabling, etc. Perhaps, after several monthsof research, projects that were counted on as significant remod-eling projects to reposition the firm’s technology platform arebetter characterized as enabling projects that are providing gen-eral enhancement of the knowledge base of the staff but notlikely to have the impact associated with a remodeling project.

A principal use of the profile and map is to support man-agement decisions across multiple funding cycles to continue orterminate projects—or make major modifications to increase thechances for successful outcomes. The changes in the map overthe years provide visual support to the year-to-year evaluation ofthe progress of the R&D program. For example, managers maymore easily recognize from the profile and map that the pres-ence of multiple reinforcing projects in one year should prompta discussion about the need to solicit a broader-based enablingproject in the coming year.

C. Experience Using the Profile

The work program positioning profile emerged from profes-sional practice of making decisions about funding R&D propos-als. At the time the profile was crafted, cost and risk were themost prominent dimensions being used to characterize R&Dproposals. The cost dimension was manifested in the need tomeet the constraint of the annual R&D budget. The Chief Engi-neer, guided by the reviews of senior technical staff members,made recommendations for funding R&D proposals. A primaryconcern about each annual budget was the distribution of fundsbetween prior year projects and new projects. So the simplestcost profile of R&D projects was to show the breakdown (bothin terms of number of projects and of dollar amount) of thosethat were newly funded, or in their second year, third year, etc.,of funding. In light of rapidly changing technology, the ChiefEngineer and the advisory team wanted to allow a portion ofthe budget every year to fund the new ideas that emerged thatyear.

The consideration of risk was a principal subject of discussionamong the senior technical staff as they evaluated proposals. Thefocus was on technological risk: the nature of the scientific andtechnical advances that were being proposed, and the likelihoodof such advances. New R&D proposals as well as the currentportfolio of R&D projects were profiled on the basis of theirtechnological risk. So there was an attempt to understand if theportfolio was, for example, very heavily funded on high-riskprojects that had a low probability of success. If there were aconscious decision to fund many high-risk projects in the hopesthat one or a few may be successful, at least that decision wouldhave been based on a thoughtful consideration of the risk. So asimple profile was the allocation of proposals along a continuumfrom low risk to high risk.

It was in this R&D proposal decision-making environmentwith its primary concerns being cost and risk that the workprogram positioning profile was developed. The profile wasmotivated by a question that is often considered in technologi-

cally based organizations: How does the R&D program relate toenhancing support to customers? This consideration of the rela-tionship of R&D proposals to the customer work program wastaken up in meetings of the Chief Engineer and senior technicalstaff as they evaluated proposals. Proposal-evaluation discus-sions served to identify various relationships between proposedR&D projects and the possible effects of those projects on thesystems engineering work program if the projects were success-ful. Instead of being vague or indifferent to the possible impactof R&D on the work program, this question was now beingconsidered explicitly.

As an example, the systems engineering practice of the com-pany included work in system performance evaluation. A pro-posal for an R&D effort in this area may be to explore thissubject at more foundational levels than the way it is encoun-tered in the everyday practice of supporting customers’ projects.A successful R&D project may result in a greatly enhanced per-formance evaluation capability and one that may distinguish thecompany from competitors. This type of R&D project evolvedinto an instance of what became the enabling (do-a-better-job)category in the work program positioning profile. Similar con-siderations of R&D proposals and their relationships to the workprogram helped to crystallize the other categories that comprisethe profile. The profile was used to guide investments in R&Defforts that were instrumental in tackling very challenging sys-tems engineering projects involving the redesign of airspaceinclusive of multiple airports, the increase in airport operationsrate capacity, and the innovative application of digital signalprocessing.

Use of the work program positioning profile each year drewattention to the possible changes over time in the relationshipsof R&D to the work program. For example, if there were con-tinued investment in a particular remodeling project year afteryear, should not the investment in “getting-smart” have maturedso that a targeting (“get-a-job”) R&D project should now be pro-posed? These longitudinal considerations motivated the plottingof possible transitions of projects from one category to anotherin the portfolio map of Fig. 1.

Assessing the R&D projects over time revealed many dif-ferent transitions, as well as R&D investments made in oneyear and never pursued. At the other extreme is a real in-stance of a lengthy transition, from remodeling to enabling,clockwise around the portfolio map in Fig. 1. This investmentwas motivated by a perceived need to get-smart in softwarequality because of the ruinous effects of poor-quality softwareon operational computer-based systems. The first R&D projectwas a remodeling one and led to published papers on new ba-sic models that explained the effects of software design de-cisions on resulting software quality [19]. Continuing R&Dinvestments over successive years culminated in an enabling(“do-a-better-job”) project that produced a design assessmentservice capability that was applied to customers’ systems de-signs to catch precursors of poor quality before implementation[20].

The experience of using the profile was that it shed light on avaluable dimension of the R&D funding; namely, the relation-ship of R&D projects to the company’s work program. As will

348 IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 56, NO. 2, MAY 2009

be further explored in Section III, there can be a wide range ofcriteria to bring to bear in making R&D investment decisions.The claim here is that the relationship of R&D to the work pro-gram merits consideration that can be enhanced by the use ofthe work program positioning profile.

D. Recommendations on Applying the Profile

Four recommendations are offered in applying the profileannually to new R&D proposals and to the entire R&D portfolio.

1) Planning: Based on the company’s mission and goals,technological competence, appetite for risk, and competi-tive environment, specify a desirable distribution of fundsto the four categories in the profile. At each annual budgetcycle, predict the transition from one category to anotherthat you expect by the next year.

2) Evaluation: Profile the current portfolio; i.e., specify theallocation (by number of projects and by funds) among thefour categories. Compare the current profile to the desiredprofile. Compare the actual transitions of R&D projects towhat was predicted. Use the results of this comparison tomotivate a discussion about the execution of the project,the perceived difficulty of the project, the advances madein the technological community, etc.

3) Proposal assessment: Assign each proposal to one of thefour categories (or a combination of them). Compare thecollection of proposals to the desired distribution of pro-posals among the four categories. Assess the extent towhich the proposals in each category relate to the objec-tives of the company, for example, do the get-smart pro-posals call for remodeling in areas that align with com-pany goals? Do the reinforcing proposals truly addresswork program areas that need support?

4) Reflection: At least annually, revisit the desirable distri-bution of R&D funds to categories based on changes intechnology, competitive landscape, goals of the company,outcomes of R&D projects, etc.

The work program positioning profile will now be placed incontext along with other profiles of R&D projects.

III. BSC AS A FRAMEWORK FOR MULTIPLE PROFILES

OF R&D PROJECTS

The work program positioning profile is not proposed assomething to be used in isolation, but rather as one additionalway to profile the R&D portfolio. To understand any portfolio,it can be helpful to employ multiple perspectives, profiling thearray of projects in various ways that are meaningful and impor-tant to the firm. As noted earlier, employing multiple profilesis consistent with the inherent multidimensionality of the R&Dactivity and the breadth of the systems engineering field.

A BSC can be useful as a framework to understand howvarious project profiles can contribute to managing R&D pro-grams. The original BSC [21] has been applied and adoptedwidely across for-profit organizations, not-for-profit organiza-tions, and government as a management and measurement struc-ture that is tied to an organization’s vision and strategy. Withthis widespread use of BSC (e.g., 50% of the Fortune 500 com-

Fig. 2. R&D BSC from [10], annotated with supporting profiles.

panies claim to use it [10, p. 44]), it is no surprise that it has beenapplied to R&D programs [22], [23]. What may be surprising isthat, with a few exceptions (noted in [10, p. vii]), BSC “remainsunder-utilized and appreciated in R&D” [10, p. vii] and has notmet with as much success as expected when used for R&D.Indeed, an extensive research investigation, employing surveys,interviews, and case studies, was conducted to focus explicitlyon the lack of success in applying BSC in R&D settings [10].

The BSC used here as a framework is the one developedby Osama [10] because it was a conscientious effort to createa “generic” BSC for R&D, based on extensive research andexamination of other proposed R&D BSCs. The Osama GenericBSC is shown in Fig. 2, encompassing the R&D mission andthe following five perspectives.

1) Customer satisfaction: Here the customers may includeinternal ones, external ones, higher level management,and funding sources.

2) Employee morale and creativity: This perspective con-siders how the R&D activity relates to reward systems,career development, and the recruitment and retention ofprofessional staff members.

3) Financial control and performance: This perspective re-lates to the effective and efficient use of resources by theR&D activity.

4) Innovation management: This is closest to the internalbusiness process perspective of classical BSC [21], in-cluding the management and processes associated withthe R&D activity.

5) Organizational learning, dissemination, and knowledgemanagement: This perspective considers how well theR&D activity benefits from lessons learned, capitalizes onthe knowledge base, and contributes to corporate memoryand institutional learning.

Profiles can serve as characterization metrics to help under-stand how the R&D activity is performing relative to each of thefive perspectives and to the R&D mission, as shown in Fig. 2.Admittedly, the association of particular profiles with specific

AGRESTI AND HARRIS: PRACTICAL PROFILES FOR MANAGING SYSTEMS ENGINEERING R&D 349

perspectives is not always clear-cut; however, the BSC perspec-tives can still provide a useful organizing structure for the manyprofiles that exist.

The profiles presented in this section have diverse origins.Where the profiles have been presented in published material,references are provided. Where there are no references cited,the profiles are ones that have been used by the authors or arebeing suggested here by the authors as candidates for use to spansome of the many dimensions of systems engineering R&D. Theprofiles are presented as they may be viewed as contributing tothe R&D mission and each of the five perspectives.

1) R&D mission:a) Multicriteria project score—Firms develop ways to

assess R&D projects using checklists or a moregeneral scoring process [12] across multiple ques-tions and factors. For example, multicriteria meth-ods like the Analytic Hierarchy Process can be usedto score each project and thereby create a dimen-sion of project value incorporating several criteriarelated to attractiveness of the project, its cost, risk,likely success, value creation potential [3], relation-ship to goals of the R&D program and the largerorganization, and other factors [l2].

b) Strategic fit/corporate priorities—The number ofprojects and dollars are matched against categoriesrelated to the strategic initiatives of the firm. Thisdimension emphasizes the connectedness of R&Dwith the firm’s strategy.

2) Customer satisfaction:a) Customer/market span—R&D project investments

are easier to justify when the project supports manycustomers or markets of the firm, not just one. In-dicators associated with this dimension are the per-centage of work program (measured, e.g., as thenumber of customers/markets, number of work pro-gram activities, or the amount of revenue) potentiallyinfluenced by an R&D project and the extent of thatinfluence. By addressing the underlying technolo-gies associated with multiple customers and mar-kets, the impact of the project on the firm’s workprogram and on customers’ operations and systemsis potentially increased.

b) Time to impact—Is the project considered basic(foundational) or applied, in terms of the “distance”between the R&D project and any realization of itsresults as actual products or services, with appliedprojects having clearer linkage to activities in thesystems engineering work program. Will the prod-ucts, outcomes, or artifacts from the research beavailable in the short term or long term; or whatis the expected year of getting a significant return(e.g., short term as one to three years; medium termas three to five years; long term as more than fiveyears). The firm’s financial health also influencesthis factor. In more challenging financial times,firms may want a greater fraction of R&D projectsaimed at short-term payoffs. Based on the funding

of the project and the planned pace, results maynot be generated in sufficient time to have the im-pact desired; the project may be overtaken by events(e.g., by a market shift or a new technology) oroutpaced by better funded or faster paced effortselsewhere.

3) Employee morale and creativity:a) Workforce development—If the R&D portfolio is

concentrated in only a few of the principal tech-nology areas of the firm, this fact can be perceivedpositively or negatively. As a positive, the clusterof projects in a given technology area can deepenthe knowledge base in that area and demonstratethe corporate dedication to enhancing its expertisein that subject. When discretionary funds are lim-ited, concentrating the investments in a few areasmay be the best strategy to attain a critical massof resources to make any kind of progress and tobe consistent with stated core competencies of thefirm. A contrarian view from this perspective of em-ployee morale is to consider how this focus of fundsmay be perceived by applied research professionalsin the technology areas not selected for the R&Dinvestment. R&D may have been playing a valuable“tool-sharpening” role for the firm’s professionalstaff, who could always feel that they had opportu-nities to get the R&D support for their good ideasto strengthen their technology areas. A concentratedportfolio may be discouraging to those in the unse-lected areas, manifesting itself in reduced employeejob satisfaction and increased turnover of key tech-nology professionals.

4) Financial control and performance:a) Resources—This factor focuses on one side of the

investment decision—the expected dollars requiredby the project, plotted over time, often by quartersand years. For rough profiling, required resourcesare sometimes characterized qualitatively in variousordinal categories that are useful for the organizationbased on the relative amount of resources, such asmajor and minor projects, or low, medium, and highinvestment projects.

b) Return on investment—In this category are a widerange of financial measures to estimate expectedreturns against investments over time, such as netpresent value, internal rate of return, profitabilityindex, and payback period [24]. It is important tomoderate the excitement that may come from cal-culations promising high returns, because they areall sensitive to the confidence that the firm has inthe anticipated incoming cash flows in future years,in the forms of cost savings or revenue streams.Consider combining this profile dimension with theone dealing with risk, to use risk-adjusted rates ofreturn [see, e.g., [24] to deal with the reality thatnot all R&D projects go on to generate positivereturns.

350 IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 56, NO. 2, MAY 2009

c) Risk—The array of R&D projects should be profiledon the basis of the degree of risk involved: how likelyis the project to be successful? A traditional view ofa portfolio would seek a spectrum of projects of low,medium, and high risk. But, the desired distributionacross these three categories can vary with the healthof the firm—for example, if the firm’s finances arenot strong, it may want more “sure things” —morelower risk projects from its R&D, or, alternatively,feel that it is even more important to invest stronglyin higher risk projects that contribute to reposition-ing the firm. This factor can be further divided intotechnical risk or program risk.

d) Success—This profile can be very useful becauseit focuses attention on precisely what will consti-tute success for each project. The related profilecan characterize the likelihood of success (e.g., aslow, medium, high) or categories related to the na-ture of success (e.g., commercial product, improvedresearch infrastructure, new competencies, imple-mented into customers’ systems and processes).

e) Uncertainty—While risk is a familiar discrimina-tor on which to profile R&D projects, uncertaintyis a related but different concept. Unlike risk, un-certainty reflects the reality of not always beingable to associate a probability of success with aproject [24, p. 470]. The uncertainty may be char-acterized in ordinal categories of low, medium, orhigh, or, for example, what may be very relevant,a two-dimensional map of projects by technical un-certainty versus market uncertainty [5]. Another po-tentially very useful treatment of uncertainty is toengage in scenario planning as a way to evaluate thelikelihood of progress or impact from your R&Dprojects. Consider “what-if” scenarios based on ac-tivities and events that are relevant to each R&Dproject. For example, how would the project be af-fected by actions of other competitors or technologyproviders, by likely advances in key technologies,by new laws, by new compliance requirements, orby the emergence of de facto standards, practices, in-frastructures, or platforms? Uncertainty assessmentshould also include consideration of the effect onR&D projects of particular “wild cards,” i.e., play-ing out nonlinear scenarios, such as the impacts ofnatural disasters, rapid acceleration of progress inquantum computing, new regulations, or politicalinstabilities in various countries.

5) Innovation management:a) IP potential—Several ordinal categories are used,

based on the extent to which the R&D projects maybe expected to generate potentially marketable intel-lectual property, and an indication of the correspond-ing feasibility of obtaining legal protection for thatresulting IP.

b) Benefits—This factor may rate projects along someordinal range such as high, medium, low to focus

on one side of the investment decision, the benefits,which may include nonfinancial rewards such as im-pact on reputation. It is easily combined with riskto show projects plotted as they are rated in a two-dimensional space of risk and reward: Is the projecta long shot, but carries with it the prospect of a bigwin if it does succeed—a “high-risk, high-reward”project?

c) Technology span—The categories show the extent towhich the projects address various critical technol-ogy areas of the firm, for example, as used in [12].

6) Organizational learning, dissemination, and knowledgemanagement:

a) External positioning—While it may be difficult toadmit, most of the advances in systems engineeringwill not come from a single firm’s R&D program.The firm must look outside its walls and maintain acontinual environmental scan as part of its businessintelligence activity [25]. What do independent re-search, professional, academic, and analyst sourcessay about critical issues and technologies? How doesyour R&D portfolio matchup against external viewsabout what will be important technologies in thefuture?

b) Global trends—How do the technologies and is-sues addressed across the portfolio relate to globaltrends? Take the perspective of the rapidly chang-ing business climate, with international markets andsourcing, pressure to collaborate and leverage ex-pertise wherever it exists, varying political and reg-ulatory environments, and demographic shifts: is theportfolio at all responsive or consistent with thesetrends? Or, does it reflect an isolated view that willhave long term negative consequences on the firm’scompetitiveness?

c) Market-pull/technology push—What are the de-mands from customers and the marketplace for sys-tems engineering support and innovation? To whatextent does your portfolio reflect market realities?Or are your R&D projects indicative of continuing topush the firm’s safe and well established technologyexpertise: “solutions looking for problems”?

d) Top-down/bottom-up—While this may seem iden-tical to the previous profile, this one highlights theorigins of the R&D ideas. Systems engineering pro-fessionals who work closely with customers are wellpositioned to develop ideas for R&D projects. Theideas may arise from simply listening to what cus-tomers identify as problems or obstacles to success.An increasing trend is for systems engineers to workat customer sites, thereby increasing the opportuni-ties to listen and learn. The firm’s engineers may alsomake connections between observable problems andunderlying causes that could be addressed by anR&D effort. This bottom-up source of R&D ideasis contrasted with ideas coming from a top-downperspective. Managers and senior technical leaders

AGRESTI AND HARRIS: PRACTICAL PROFILES FOR MANAGING SYSTEMS ENGINEERING R&D 351

who have visibility across multiple work programsmay see opportunities for a top-down-enabling R&Dproject to investigate a crosscutting issue or techni-cal challenge—and one that a systems engineer onany single work program would not notice.

e) Work program positioning—Introduced in this pa-per, this profile highlights the relationship of theR&D projects to the ongoing work program, by lo-cating projects in the portfolio space as they areidentified as targeting, reinforcing, enabling, and re-modeling projects.

In addition to its primary role as a management and measure-ment framework, the BSC can serve as a useful way to organizethe many profiles of R&D project portfolios. The various pro-files have been shown in Fig. 2 as they support the R&D missionand the five perspectives of the R&D BSC. In particular, Fig. 2shows that the work program positioning profile introduced inthis paper fits in naturally to support the Organizational Learningperspective.

IV. CONCLUSION

There is no substitute for effective leadership of the R&D pro-gram, to provide vision, perspective, judgment, good instincts,and a constant linkage to the firm’s strategy, customers, and linesof business. In the highly competitive marketplace of systemsengineering professional services, the responsible stewardshipof a firm’s discretionary R&D investments can be a decisivefactor in its success. With the abundance of advice availableon managing R&D portfolios, systems engineering firms areencouraged not to overlook the potential benefits from verypractical profiles, including the one introduced here to clarifythe relationships of R&D projects to work programs.

ACKNOWLEDGMENT

The authors thank anonymous referees and editors for manyconstructive suggestions to improve the structure of this paper.

REFERENCES

[1] R. G. Cooper, S. J. Edgett, and E. J. Kleinschmidt, “Best practices formanaging R&D portfolios,” Res. Technol. Manage., vol. 41, pp. 20–33,Jul./Aug. 1998.

[2] K. Heidenberger and C. Stummer, “Research and development projectselection and resource allocation: A review of quantitative modeling ap-proaches,” Int. J. Manag. Rev., vol. 1, pp. 197–224, Jun. 1999.

[3] J. D. Linton, S. T. Walsh, and J. Morabito, “Analysis, ranking, and selectionof R&D projects in a portfolio,” R&D Manage., vol. 32, no. 2, pp. 139–148, 2002.

[4] R. Seider, “Optimizing project portfolios,” Res. Technol. Manage., vol. 49,pp. 43–48, Sep./Oct. 2006.

[5] I. C. MacMillan and R. G. McGrath, “Crafting R&D project portfolios,”Res. Technol. Manage., vol. 45, pp. 48–59, Sep./Oct. 2002.

[6] A. Ali, M. U. Kalwani, and D. Kovenock, “Selecting product developmentprojects: Pioneering versus incremental innovation strategies,” Manage.Sci., vol. 39, pp. 255–274, Mar. 1993.

[7] R. Khorramshahgol, H. Azani, and Y. Gousty, “An integrated approachto project evaluation and selection,” IEEE Trans. Eng. Manage., vol. 35,no. 4, pp. 265–270, Nov. 1988.

[8] C. Stummer and K. Heidenberger, “Interactive R&D portfolio analysiswith project interdependencies and time profiles of multiple objectives,”IEEE Trans. Eng. Manage., vol. 50, no. 2, pp. 175–183, May 2003.

[9] R. L. Schmidt and J. R. Freeland, “Recent progress in modeling R&Dproject-selection processes,” IEEE Trans. Eng. Manage., vol. 39, no. 2,pp. 189–201, May 1992.

[10] A. Osama, “Multi-attribute strategy and performance architectures inR&D,” Ph.D. dissertation, Frederick S. Pardee—RAND Graduate Schoolfor Policy Studies, Santa Monica, CA, Mar. 2006.

[11] J. Malek, “The path to smart R&D,” Pharma. Exec., vol. 23, pp. 70–80,Nov. 2003.

[12] L. Canez and M. Garfias, “Portfolio management at the Mexican petroleuminstitute,” Res. Technol. Manage., vol. 49, pp. 46–55, Jul./Aug. 2006.

[13] A. P. Sage, Systems Engineering. New York: Wiley, 1992.[14] (2007, Dec. 5). What is Systems Engineering?, International Council on

Systems Engineering [Online]. Available: http://www.incose.org.[15] (2007, Dec. 5). Possible Research Topics, Systems Engineering Center

of Excellence, International Council on Systems Engineering [Online].Available: http://www.incose.org.

[16] (2007, Mar. 5). R&D 100, IEEE Spectrum Online, Dec. 2006 issue[Online]. Available: http://www.spectrum.ieee.org/dec06/rd100.

[17] R. F. Bordley, “R&D project selection versus R&D project generation,”IEEE Trans. Eng. Manage., vol. 45, no. 4, pp. 407–413, Nov. 1998.

[18] N. Radjou, “Does corporate R&D still matter?,” Res. Technol. Manage.,vol. 49, pp. 6–7, Jul./Aug. 2006.

[19] W. W. Agresti and W. M. Evanco, “Projecting software defects fromanalyzing Ada designs,” IEEE Trans. Softw. Eng., vol. 18, no. 11, pp. 988–997, Nov. 1992.

[20] W. W. Agresti, “Profile of an artifact assessment capability,” in Experi-mental Software Engineering Issues, D. Rombach, V. Basili, and R. Selby,Eds. Berlin, Germany: Springer-Verlag, 1992, pp. 13–16.

[21] R. S. Kaplan and D. P. Norton, “The balanced scorecard: Measures thatdrive performance,” Harvard Bus. Rev., vol. 70, no. 1, pp. 71–79, Jan./Feb.1992.

[22] W. G. Bremser and N. P. Barsky, “Utilizing the balanced scorecard forR&D performance measurement,” R&D Manage., vol. 34, no. 3, pp. 229–238, 2004.

[23] G. B. Jordan and J. C. Mortensen, “Measuring the performance of researchand technology programs: A balanced scorecard approach,” J. Technol.Transfer, vol. 22, no. 2, pp. 13–20, Jun. 1997.

[24] H. C. Petersen and W. C. Lewis, Managerial Economics, 4th ed. UpperSaddle River, NJ: Prentice-Hall, 1999.

[25] J. Liebowitz, Strategic Intelligence: Business Intelligence, Competitive In-telligence, and Knowledge Management. New York: Auerbach/Taylor &Francis, 2006.

William W. Agresti (SM’88) received the B.S.degree from Case Western Reserve University,Cleveland, OH, and the M.S. and Ph.D. degrees fromNew York University, New York.

Previously he held senior technical positions atMitretek Systems Inc., MITRE Corporation, andComputer Sciences Corporation. He was also the Pro-gram Director for Experimental Software Systems inthe Computer and Information Science and Engineer-ing (CISE) Directorate, National Science Foundation(NSF). He is currently a Professor at Carey Busi-

ness School, Johns Hopkins University, Baltimore, MD. His current researchinterests include systems engineering management, software engineering, infor-mation security management, and data mining. He is a member of the EditorialBoards of Empirical Software Engineering and the Encyclopedia of SoftwareEngineering.

Richard M. Harris (SM’90) received the B.S.E.E.and M.S.E.E. degrees in operations research fromMassachusetts Institute of Technology, Cambridge,and the Ph.D. degree in engineering economic sys-tems from Stanford University, Palo Alto, CA.

He was the Vice President for the Center forAdvanced Aviation System Development (CASSD),MITRE Corporation, where he was also a Chief Engi-neer for Washington Operations, and Director of theNaval Systems Engineering Division. He is currentlyretired and is an independent Internet consultant.

Dr. Harris is an Associate Fellow of the American Institute of Aeronautics(AIAA).