Forecast articles from TDWI’s Directors of Education and Research

Volume 2

Featuring the very bestof TDWI’s BI articles,research summaries,and newsletters

Ground-breaking research on:• Business Performance

Management• Consolidating Analytic Silos• BI Strategies for

CRM/ERP Data

Insights and perspective from

BI industry experts

Introducing BusinessObjects XI.

eXtreme Insight. Trusted Platform.

You know the problem. Information you can’t find, trust, or understand. Get to know the solution.

BusinessObjects™ XI, our breakthrough business intelligence (BI) suite. It provides easy access to timely,

accurate information, and everything you need to turn that information into better decisions. All on the

most trusted BI platform. BusinessObjects XI. See the extreme difference it can make.

www.businessobjects.comTo learn more about BusinessObjects XI, or to register for our eXtreme Insight World Tour, visit

BusinessObjects XI eXtreme Insight World Tour Amsterdam • Beijing • Birmingham • Brussels • Chicago • Dublin • Madrid • Melbourne

Milan • Munich • New York • Paris • Rome • San Francisco • Seoul • Shanghai • Singapore • Sydney • Stockholm • Washington DC • Zurich

The B

usi

ness

Obje

cts

logo a

nd B

usi

ness

Obje

cts

are

tra

dem

ark

s of B

usi

ness

Obje

cts

S.A

. A

ll oth

er

nam

es

refe

renced h

ere

in m

ay

be t

he t

radem

ark

s of th

eir r

esp

ectiv

e o

wners

. ©

20

05

Busi

ness

Obje

cts

S.A

. A

ll rights

rese

rved.

Project11 1/28/05 3:58 PM Page 1

1VOLUME 2

2005 Forecasts from the Directors 4 Business Intelligence: 2005 and Beyond

Dave Wells

6 Seeds of ChangeWayne Eckerson

2004 Report Series9 Best Practices in Business Performance Management—Excerpt

Wayne Eckerson

13 In Search of a Single Version of Truth—ExcerptWayne Eckerson

17 Developing a BI Strategy for CRM/ERP Data—ExcerptColin White

TDWI FlashPoint22 Architectural Alternatives for Data Integration, Parts I & II

Evan Levy

Ten Mistakes to Avoid Series27 Ten Mistakes to Avoid When Selecting and Deploying BI Tools

Cindi Howson

33 TDWI Educational Programs

BI This Week 34 Making the Case for XML/A

Stephen Swoyer

Business Intelligence Journal 37 BI Case Study: Ford’s Business Assistance Center Hits the

Fast Lane with IlumifiAPFMark Hammond

Case Studies & Solutions41 It Takes Two to Tango

Wayne Eckerson

Salary Survey45 TDWI 2004 Salary, Roles, and Responsibilities Report Summary

49 About TDWI

Table of Contents

Sponsor Index

Baseline Consulting

Business Objects SA

DataMirror

Hyperion SolutionsCorporation

Informatica Corporation

MicroStrategy, Inc.

Sunopsis Inc.

01_TOC_r030105 3/1/05 12:00 PM Page 1

We are pleased to present to you our second annual TDWI’s Best of Business

Intelligence: A Year in Review. A decade ago, TDWI was formed to provide data ware-

housing professionals a single source of information to help them do their jobs more

effectively and stay current with the rapidly developing industry in which they worked.

As we enter our second decade of business, we continue to provide the highest-quality

business intelligence (BI)and data warehousing education and research in the industry.

During the past year we have continued to focus on BI and data warehousing educa-

tion, content, and information through our conference and seminar curriculum, onsite train-

ing, publications, research, e-newsletters, and CBIP certification. As part of our effort to

continue to educate our audience, we are happy to present the second volume of Best of

Business Intelligence: A Year in Review.

For this publication we have selected a sampling of our best, most well-received,

impact-full articles, research, and information. Highlights include selections from our

exclusive TDWI Membership publications, including the 2004 Salary, Roles, and

Responsibilities Report, Business Intelligence Journal, and our Ten Mistakes to Avoid

series. Additionally, look for excerpts from our Report Series (on topics such as BPM,

Consolidation, and CRM/ERP) and from our e-newsletters: TDWI Flashpoint, BI This Week,

and TDWI’s Case Studies & Solutions.

You will also find forecast articles from Wayne Eckerson, TDWI’s director of research,

and Dave Wells, TDWI’s director of education. These articles offer a pertinent comparison

of their predictions for 2004 to their predictions for 2005—see how close to the mark they

were and gain insight for the coming year.

TDWI is committed to providing industry professionals with information that is educa-

tional, enlightening, and immediately applicable. We look forward to your feedback on the

Best of Business Intelligence, Volume 2. We would also like to thank the authors for their

willingness and dedication to sharing their industry knowledge and expertise.

Enjoy!

Denelle Hanlon

Editorial Director, TDWI’s Best of Business Intelligence

TDWI

dhanlon@tdwi.orgTDWI is a division of 101communications LLC basedin Chatsworth, California.

Editorial Director Denelle Hanlondhanlon@tdwi.org

Production Editors Theresa Johnstontjohnston@tdwi.org

Jennifer Ageejagee@tdwi.org

General Manager Peter Quinn

Director of Education Dave Wells

Director of Research Wayne Eckerson

Director of Marketing Meighan Berberich

Art Director Deirdre Hoffman

5200 Southcenter Blvd., Suite 250Seattle, WA 98188206.246.5059 Fax: 206.246.5952info@tdwi.org, www.tdwi.org

Advertising Opportunities: Denelle Hanlon, 5200Southcenter Blvd., Ste. 250, Seattle, WA 98188.206.246.5059 ext. 102, dhanlon@tdwi.org.

Reprints: For editorial and advertising reprints of100 copies or more, and digital (Web-based) reprints,contact Reprint Management Services atwww.reprintbuyer.com or Jennifer Hartman at717.399.1900, jhartman@reprintbuyer.com.

List Rentals: For e-mail list rentals or our 101directmasterfile, contact Cathy Dodies, 215.643.8047,cdodies@101com.com or Jane Hope, 215.643.8046,jhope@101com.com. For postal list rentals contactour list manager at Statlistics: Dolores Broderick at203.778.8700.

Corporate Headquarters: 9121 Oakdale Ave. Ste.101, Chatsworth, CA 91311, www.101com.com.

©2005 by 101communications LLC. All rightsreserved. Reproductions in whole or part prohibitedexcept by written permission. Mail requests to“Permissions Editor,” c/o Best of BI, 2005, 5200Southcenter Blvd., Ste. 250, Seattle, WA 98188. Theinformation in this magazine has not undergone anyformal testing by 101communications LLC and is dis-tributed without any warranty expressed or implied.Implementation or use of any information containedherein is the reader’s sole responsibility. While theinformation has been reviewed for accuracy, there isno guarantee that the same or similar results may beachieved in all environments. Technical inaccuraciesmay result from printing errors, new developments inthe industry, and/or changes or enhancements toeither hardware or software components. Printed inthe USA.

TDWI is a trademark of 101communications LLC.Other product and company names mentioned here-in may be trademarks and/or registered trademarks oftheir respective companies.

Manufacturing & Distribution Director Carlos Gonzalez

Production Manager Tanya de Somov

Chairman Curt Hessler

President & CEO Jeffrey S. Klein

Executive Vice President Gordon Haight

Senior Vice President & CFO Stuart K. Coppens

Senior Vice President,CIO & General Counsel Sheryl L. Katz

Senior Vice President,Operations Brad Stauffer

Senior Vice President,Human Resources Michael J. Valenti

Vice President, Business Technologies Group Ellen Hobbs

2

Volume 2, March 2005

Gain a unified customer view. Improve customer

service. Enable real-time data processing. Ensure

data integrity. Maintain regulatory compliance.

Create operational efficiency.

Reduce costs by hundreds of thousands A world-class leader in financial products and services did it.

(With some help from Informatica)

Find out how you can too.

Visit www.informatica.com/info/tdwiadq105 to read more about this customer's

success, and learn how Informatica has helped leading organizations worldwide

meet their end-to-end data integrations needs.

© 2005 Informatica Corporation. All rights reserved. Informatica and the Informatica logo are trademarks or registered trademarks of Informatica Corporation.

Project1 2/7/05 9:56 AM Page 1

4 TDWI’S BEST OF BUSINESS INTELLIGENCE

FORECAST

ooking to the future is alwaysinteresting and challenging.Attempting to guess what the

future holds is either a fool’s errand oran element of planning and direction-setting. I like to think of it as the latter,but looking back at predictions of thepast may be the best way to decide. As2003 drew to a close, I made severalforecasts about the shape of BI in 2004.One year later, it’s time to revisit thosepredictions and to offer some new onesfor the coming year.

Looking Back—2004 A consistent theme through the 2004 predictions was a shift from tech-nology-focused to organizationallyfocused BI. I stated that “although tech-nology will have a role, the most mean-ingful trends will be non-technical.” I predicted several innovations in busi-ness management and informationmanagement “that will have lastingeffects well beyond the end of the com-ing year.” Those predictions included:

Prediction 1: From Data Warehousingto Business Intelligence. BI program management will expand toencompass change management, quality

management, data governance, processand workflow alignment, and more.

Result: A good beginning but muchleft to be done. Most companies areseeking to make the transition fromdata warehousing (a data integrationactivity) to BI (an information servicesactivity). Alignment of BI programswith business strategies is a dominanttheme, and data governance has gainedstrong presence in much of the BI com-munity. Other key factors in the transi-tion, however, lag behind. The need forchange management and quality man-agement are recognized by many, butpositive actions for meeting that needare far from commonplace. I expectthese areas to expand in 2005.

Prediction 2: Driving to Value.Financial caution is the order of theday, and value received is the ultimateconsideration to justify continued BIinvestment. Many companies willfocus on real business impact as themeasure of BI value.

Result: Right on target. Even as theeconomy rebounds, financial prudenceremains at the forefront. The past year

has been one of cost-conscious BI.Techniques to build the business caseand to measure BI cost and value areamong the most popular of TDWIeducation programs. Many softwarevendors have embraced the valuetheme, marketing products and build-ing partnerships with key customerswith a business value perspective.Expect this trend to continue and togrow in 2005 and beyond.

Prediction 3: From Data Integrators toInformation Architects.Information technology (IT) organiza-tions will develop new skills and tech-niques in response to changing businessvalue expectations. Business processesand process-based management disci-plines become the foundation for BI requirements.

Result: Barely scratched the surface.We’ve seen some small steps in thedirection of becoming business-orient-ed architects of information, but it is far from becoming a widely prac-ticed discipline. Methodologies areemerging to recognize the need (TheBI Pathway from DecisionPath, forexample) but methodological change is

BusinessIntelligence: 2005 and Beyond

by Dave Wells

L

only a first step. The essential nextsteps involve building a community ofpractice around new, business-focusedapproaches. Most technical profession-als still have only a limited, and tech-nology-focused, understanding of busi-ness management disciplines such asBPM, SCM, and CRM. There’s a longjourney ahead of us in order to becomebusiness-savvy IT people, and to evolvedata modeling methods to better repre-sent integrated business metrics.Expect continued progress through2005, but it is likely to be 2006 andbeyond when these practices becomecommonplace.

Prediction 4: From Integrated Data toActionable Information.The shift to information architecturedrives changes in the way that we thinkabout data, and in the way that wethink about integration. Leading BIorganizations will view unstructureddata as equal in value to structureddata. They will embrace un-trusted dataas an opportunity to enrich the contextand enhance the value of trusted data.They will break down the boundarybetween operational and informationalsystems as a primary way to make infor-mation become actionable.

Result: Still waiting for this one. Theoverwhelming majority of today’s datawarehouses still depend solely on oper-ational data sources and structureddata. For most business people, gather-ing enough decision-making informa-tion for confident actions remains achallenge. Desired information is wide-ly scattered in a variety of locations andformats, including data warehouses,scorecards and dashboards, Webresources, and much more. Collectingand correlating information continuesto be difficult and time consuming. Wehave yet to achieve the goal of integrat-ed business information services. Realprogress here depends on continuedadvancement toward becoming infor-mation architects (prediction 3).Expect slow progress here through

2005, with the pace picking up as wenear the end of the year.

Prediction 5: Organizational Change—Becoming a BI Organization.The new BI culture will blend businessand IT knowledge and skills in a singleorganization. Business leaders willbecome more IT-savvy; IT leaders willbecome more business-savvy. This neworganization will unite business skillsand technical skills in the same organi-zation, and at the leadership level in thesame individuals.

Result: Mixed and widely variant.Organizational and cultural changes areperhaps the most challenging of all. Afew leading organizations are makingreal strides toward advanced BI cul-tures. Many others continue to workwith old patterns, and some have yet torecognize the need for change. Thoughtleaders in the industry (e.g., Kaplanand Norton with the Strategy Mapsconcept) continue to drive the change,but cultural evolution is slow. I expectbusiness/IT working relationships to bea predominant theme throughout2005. Improved understanding andworking relationships, however, areonly the beginning. The shift tobecome true BI organizations will takeplace over the next several years.

What to Expect in 2005Anticipate progress in all of the areasdescribed above. Change managementand quality management will continueto grow as key disciplines. Valuationmethods and metrics will standardizeand become more widely used.Communities of practice will emergearound business-oriented informationarchitecture. Slow but steady progresswill be made toward integrated informa-tion services. Many will begin to narrowthe business/IT gap and take the earlysteps to becoming a BI organization.

The coming year, however, offers morethan continued progress in knownfields. It will introduce new topics,

present new challenges, and seed inno-vations of the future. Among the newthemes for 2005 I forecast:

BI Meets KM:The fields of business intelligence andknowledge management (KM)willbegin to converge. As BI reaches out toinclude unstructured information, thefundamental concepts and techniquesare found in the disciplines of KM. Inmuch the same way that data miningdisciplines provide the foundation ofpredictive analytics, KM discipline isfundamental to integrated informationservices. Managing business rules andcollective business knowledge as inte-grated pieces of a shareable informa-tion resource extends the reach andenhances the value of business intelli-gence systems.

Personal Information Services:Demand for personalization of BI services increases as the scope and useof BI systems expands. BI tools andtechnologies begin to include featuresand functions to connect informationwith unique and individual knowledgeof the people who use that informationthrough personalization and mass customization.

Visualization Skills Gain Traction: Developers of BI systems begin to rec-ognize the importance of graphics andpresentation skills. Leading organiza-tions begin to evolve their BI solutionsfrom information systems to businesscommunication systems. Successfulcommunication systems demand thatwe distinguish between accurate graphics and enlightening graphics.Visualization skills become a key element to providing new businessinsights through information services. n

5VOLUME 2

FORECAST

Dave Wells is Directorof Education at TDWI.He can be reached atdwells@tdwi.org.

6 TDWI’S BEST OF BUSINESS INTELLIGENCE

FORECAST

s any good futurist knows, theseeds of future change aresown in the present. The dom-

inant forces of tomorrow are visibletoday in the backwaters and eddies ofexisting markets and communities.

Last year at this time, I made seven predictions about major events ortrends that would crystallize in 2004. Ithought it might be interesting to seehow well my forecasts held up in thecrucible of reality.

Prediction 1: BI versus ERP.Leading independent BI vendors willcontinue to bulk up, buying marketshare to counter serious threats fromERP vendors entering the BI market.

Result: Accurate in concept. After themega BI acquisitions engineered byBusiness Objects and Hyperion in2003, not much happened in 2004,probably because there is little left togobble up. IBM did acquireAlphaBlox, and Informatica acquiredStriva, but these are minor instances.

Nevertheless, ERP vendors continue tomake inroads into BI with their pack-aged offerings. ERP customers who

Seeds of Changeby Wayne Eckerson

A

7VOLUME 2

FORECAST

have made substantial commitments tothe ERP software find it a no-brainerto use the vendor’s analytical softwarefor all their BI needs. This is certainlytrue with SAP, which has successfullyconverted many of its global customersto SAP BW 3.x (now ready for primetime). Siebel Systems has gained sub-stantial traction as a pure-play BIvendor, although the bulk of its salesstill comes from Siebel CRM cus-tomers. Oracle BI tools are being usedby many customers, especiallymedium-sized firms with enterpriseOracle contracts. Why buy a BI toolwhen you’ve already got one?

Rather than pursue acquisitions, BIvendors seem to believe that the bestdefense is a good offense. Thus, Cognosand Hyperion, in particular, areworking hard to etch themselves as theleading providers of business perform-ance management applications (i.e.,budgeting, planning, forecasting, mod-eling, and performance management).This is a smart move since analyticsplay a key role in BPM applications,and they seem to be gaining traction,although the market is not as fast-growing as they would like. BusinessObjects, for its part, has done anadmirable job of integrating CrystalSoftware’s organization and products.Although its market share is much lessthan was envisioned at the time of theacquisition, Business Objects in 2004became the undisputed market leaderin terms of revenues.

Prediction 2: The Popularity ofDashboards Explodes. Result: Dead on! Who didn’t build a dashboard in 2004? Okay, people arecalling whatever they build a dash-board, as long as it displays a few keymetrics on the opening screen.Nonetheless, the business communityhas finally discovered the medium bywhich it wants to consume analyticalinformation. A dashboard gives them aquick glance at the key measures that

they need to track and lets them drilldown into more detail if something isawry. In essence, dashboards are graph-ical exception reports. The messagefrom users is, “Don’t bother me ifnothing is wrong.” The better dash-boards proactively alert users whensomething needs attention by sendingthem an e-mail, page, or phone call.

TDWI defines three types of dash-boards: operational, tactical, andstrategic. While most people think theyare building strategic dashboards—which we define as cascading dash-boards tied to business strategy—mostare deploying tactical dashboards,which are really just glorified reportingportals. Operational dashboards areused by front-line specialists tomonitor daily business processes in“right time,” enabling them to inter-vene to fix problems or capitalize onopportunities before it’s too late.

Prediction 3: Firms Standardize onBI Tools. Results: Sounds good on paper…hard to execute. At the start of 2004,the economy continued an extremelysluggish recovery, and managers werelooking everywhere to cut costs. Manymanagers saw the plethora of BI toolsin their organizations as ripe forpicking. “Why have 15 BI tools andthe separate development, support,and training staffs to go with themwhen we can have just one tool?” thesemanagers surmised.

Although standardizing on BI tools cansave a lot of money, it is a politicalminefield. Many BI managers say it’seasier to consolidate data warehousesthan to consolidate BI tools, whichsometimes takes years. People andprocesses come to rely on specific BItools, and replacing them meansretraining users, rewriting reports, andperhaps reengineering processes—eachof which takes time, money, and polit-ical willpower.

Prediction 4: Data IntegrationExtends ETL. Results: Still waiting on this one.ETL vendors are trying to morph into “data integration” vendors.Unfortunately, they are still in thecocoon phase of their metamorphosis.Financial earnings from the top twoETL vendors, Ascential Software andInformatica, were pretty meager thisyear, with some significant downturnsalong the way.

Although I believe the data integrationmarket is real, I also know from experi-ence how hard it is to sell “middle-ware.” The data integration marketsuffers from being overly broad andeasily usurped by hordes of internalprogrammers (who now are four timescheaper because they live, for example,in India). Many organizations don’tthink about using a tool to move datafrom place to place, unless it musthappen on a regular basis to supportoperational processes. And why use anETL tool for that?

Part of my prediction claimed thatETL vendors would finally make adent in providing really usable meta-data tools. Informatica releasedSuperGlue, and Ascential has itsMetaBrokers; both are seeminglybright spots in the traditionally murkymetadata market. But metadata is atough nut to crack. There are just toomany tools from too many vendors torealistically believe a commercialproduct can integrate with all of themat a sufficient level in an automatedfashion to deliver real business value.And what executive wants to pay formetadata? What’s that?

Prediction 5: Backlash against AdHoc BI.Results: Depends on how you define“ad hoc.” Ad hoc BI is dead, at leastthe way we define it. Traditionally, adhoc BI is one step above hand-writingSQL. Ad hoc BI tools let users graphi-cally define queries using predefined

query objects (e.g., semantic layer).Our industry has finally discoveredthat graphically creating queries isreally not much better than hand-writing them. Still, only a small frac-tion of power users are inclined to use“true” ad hoc query tools.

In 2004, I discovered that BI managerswho have deployed “ad hoc” capabili-ties to their users have really deployedsomething other than ad hoc querytools. In most cases, they’ve deployedparameterized reports where users canfilter predefined reports using a seriesof predefined dropdown lists. Orthey’ve deployed OLAP views into anOLAP database, which itself creates apredefined sandbox in which users cannavigate. Most of the OLAP views aresimply glorified briefing books thatstep users through predefined views,which users can drill into if they so desire.

The bottom line is that most businessmanagers know what information theyand their subordinates ought to belooking at on a regular basis. Yes, ifsomething is out of the ordinary, theyneed to be able to explore and get moreinformation easily. But, really, whatmost users want and need is a dash-board with interactive reports under-neath—a drillable exception report, ifyou want to get down and dirty!

Prediction 6: Data Mining Surges. Results: Good so far, but don’t ask forproof just yet. I predicted that moreorganizations in 2004 would beginexploring data mining techniques as away to enhance the strategic value oftheir data warehouses. Anecdotally, Ican say with great confidence that thishas happened. At TDWI, we seegreater interest in data mining classes.We think the time is right for wholesaleadoption of data mining because theBI market has matured and organiza-tions are looking to extract more valuefrom their investments—which some-times reach into the tens of millions ofdollars. Of course, not all organizationshave a need for data mining. It’s not acheap thing to get into, so if you don’tneed it, don’t push it!

Prediction 7: Offshore BI Gains Steam.Results: Mixed. I predicted that thenumber of BI jobs farmed out to out-sourcing firms would grow from atrickle to a torrent. I had already seennumerous firms move ETL develop-ment to India and elsewhere. Whilemany BI shops take it for granted thatthey will be using offshore help, I’vealso encountered a number of man-agers who say that they tried offshoringand it doesn’t work very well in BI,which requires iterative developmentand close communication with users. Isuspect that in 2005 we’ll really begin

to discover which jobs and tasks makesense to offshore and which don’t. At amacro level, any analysis and projectmanagement must stay onshore, butpure development based on well-defined specifications can go abroad. n

8 TDWI’S BEST OF BUSINESS INTELLIGENCE

FORECAST

Wayne Eckerson is Director of Research at TDWI. He can be reached at weckerson@tdwi.org.

9VOLUME 2

BUSINESS PERFORMANCE MANAGEMENT

Growth Despite ConfusionMany people today are confused about what business performance management(BPM)is—and is not. Much of the confusion arises because BPM is a new categoryto describe multiple applications that many companies have already implemented,including planning, business intelligence, budgeting, and KPI reports. When intro-duced to the concept of BPM, many managers rightfully exclaim, “We’ve been doingthat for years!”

However, most organizations haven’t pulled these applications together in a cohesive and concerted way—using a common strategic and technical frameworkto drive all parts of the organization toward a common set of goals and objectives.

0 10 20 30 40 50 60 70 80

Best Practices in BusinessPerformance Management

Greater visibility into the business

Better execution of strategy

More efficient processes

Faster reaction to events

Better strategic planning

Single version of the truth

Better coordination among groups

Revenue/customer growth

New regulations

65%73%

53%57%

53%44%

52%53%

50%54%

49%57%

47%44%

39%34%

19%20%

BPM Desired Benefits and Drivers

Deployed BPMExploring BPM

Illustration 1. The top reason that organizations are deploying BPM is to gain greater visibilityinto the business. Based on 635 respondents.

Business performance

management is a

series of processes

and applications

designed to optimize

the execution of

business strategy.

—TDWI definition

From the TDWI Research Report Best Practices in Business PerformanceManagement: Business and Technical Strategies, sponsored by Applix, Inc.,Business Objects SA, Cognos Inc., Hyperion Solutions Corporation, andThinkFast Consulting, Inc. This report was published in March 2004.

by Wayne Eckerson

BUSINESS PERFORMANCE MANAGEMENT

10 TDWI’S BEST OF BUSINESS INTELLIGENCE

Today, companies implement individual BPM applicationsisolated from each other. Each application provides somelocal benefit, but little global value.

Benefits. Despite the current confusion, BPM is gainingadherents in all industries. Today, early adopter companiesare deploying BPM initiatives to better communicate andexecute business strategy at all levels and gain greater visi-bility into organizational performance. The success of earlyadopters of BPM is stoking red-hot interest in the area byorganizations in all industries.

According to TDWI research, almost two-thirds of organi-zations (65 percent) said they have adopted BPM to gain“greater visibility into the business,” followed by “betterexecution of strategy” and “more efficient processes” (53percent each), and “faster reaction to events” (52 percent).(See Illustration 1 on page 9.)

When all steps in the BPM process are executed in a concert-ed manner, they enhance communication, collaboration,control, and coordination among staff and groups in theorganization. In many ways, BPM greases all the parts of theorganizational engine to keep it moving in the right direction.

Enterprise BI. Successful BPM projects have a strong datawarehousing and business intelligence foundation. Datawarehouses integrate data from multiple sources to deliverkey performance indicators and detailed reports, and busi-ness intelligence tools display data in the appropriate for-mat (dashboard, scorecard, detailed report, etc.) as well assend alerts and notifications to users when performanceslips below acceptable levels.

BPM Framework: A Four-Step ProcessOrganizations are discovering that BPM is more than just aplanning, budgeting, forecasting, consolidation, or score-carding application—rather it’s a common business andtechnical framework that underlays all these applications.

The BPM framework consists of a four-step businessprocess—Strategize, Plan, Monitor, and Act & Adjust. (SeeIllustration 2.) Each step in the process is enabled by vari-ous tools, chief among them data warehouses and businessintelligence tools.

BI for BPM. As such, BPM places tremendous demands onBI and data warehouses to deliver the right information tothe right people at the right time. In essence, BPM solu-tions require a turbo-charged, enterprisewide analyticalplatform. This is a far cry from the departmental BI solu-tions that most organizations have implemented to date.

Where to Begin?Organizations embarking on BPM face several challenges.Since many organizations view BPM as an enterpriseendeavor, they struggle to figure out where to begin. Dothey deploy at the enterprise, division, or department level?To executives, mid-managers, or front-line employees?

Scope. Most organizations (72 percent) are deploying BPMon an enterprisewide basis. This is a surprisingly high per-centage compared to those organizations that are contem-plating BPM. Only 25 percent of the latter group envisiondeploying BPM on an enterprisewide level, while 24 per-cent envision departmental deployments initially. (SeeIllustration 3.)

Start Small. Whatever the approach or initial application,most BPM practitioners recommend starting small, with afocused project that can be implemented quickly yet deliv-er substantial benefits.

IntegratedData

1.St

rategize 2. Plan

3.Monitor4.Act&

Adjus

t

Strategic Plans

Strategy M

aps

Plans, Budgets

&Scenarios

Projects/Initiatives

Scorecards

Reports/A

nalysis

Alerts

Forecasts

& Models

Agents, Offers

& Actions

Strategy

Execution

Illustration 2. BPM Framework

0 10 20 30 40 50 60 70 80

Enterprisewide

Divisionwide

Departmentwide

Not sure

BPM Scope

Illustration 3. Most existing BPM deployments are enterprisewide inscope, although organizations “exploring” BPM envision smallerimplementations. Based on 635 respondents.

72%25%

23%31%

5%24%

1%17%

Deployed BPMExploring BPM

BUSINESS PERFORMANCE MANAGEMENT

11VOLUME 2

How to Create Effective KPIs?Another challenge is defining valid, relevant, and effectivekey performance indicators (KPIs) tailored to each group inthe organization. What is the right number of KPIs? Whichreally drive strategic value and proactive action? Which con-flict with other KPIs or incent employees to work at cross-purposes with key objectives? TDWI research shows thatthere are seven critical attributes of effective KPIs:

1. Standardized. KPIs contain standardized measures and attributes that are agreed on by everyone in the organization.

2. Valid. KPIs contain valid data that can be reconciled with source systems and gain the “trust” of end users.

3. Comprehensible. KPIs are easy to comprehend becauseorganizations display only a handful per user or role andtrain users on what they mean and what to do about them.

4. Contextual. KPIs provide users context to assess theirperformance by applying thresholds or benchmarks to the raw data.

5. Actionable. KPIs lead to positive action because usershave access to timely information to work in a proactivemanner.

6. Empowering. KPIs empower users because firms attachfinancial or other incentives to them, encouraging per-formance “in the right direction.”

7. Relevant. KPIs maintain their relevance because organiza-tions continually evaluate their effectiveness and refinethe metrics.

How to Architect a BPM Solution?Architecting a BPM solution can also be challenging sinceit requires pulling together data from a multiplicity ofgroups, each with different business processes and systems,to deliver a single version of truth. Even with a BI infra-structure in place, organizations can struggle to figure out how to enhance their existing infrastructure to support BPM.

The danger with BPM systems, especially packaged solu-tions that come with predefined data models, is that theycan become silos of information, fracturing a consolidatedview of corporation performance.

It appears that almost three-quarters (71 percent) of organ-izations have avoided this dilemma by pulling data directlyor indirectly from a central data warehouse, according toTDWI. (See Illustration 4.) Half (50 percent) pull datadirectly from a data warehouse, and 21 percent pull datafrom a dependent data mart connected to a data warehouse.

How to Ensure Usage?Finally, deploying a BPM solution can be half the battle;getting users to use the solution and drive performance inthe right direction often requires a cultural transformationthat must be initiated by top executives and reinforced withcarefully thought-out compensation plans and incentives.

Organizations can also improve usage by delivering KPIs ina form that users can easily consume. For example, mostusers respond positively to graphical dashboards or score-cards embedded in corporate portals. These environmentsrequire little (if any) end-user training. Keeping KPIs rele-vant through constant re-evaluation and refinement alsoensures ongoing usage.

Report ConclusionGiven the current economic climate, increased financialpressures, and the evolution of information technology, it’sno surprise that BPM is gaining traction now. It providesorganizations with a new competitive edge. It gives execu-tives a better way to communicate and monitor strategy,and empowers employees to react quickly to business eventsand better execute key strategies. n

Wayne Eckerson is Director of Research at TDWI. He can bereached at weckerson@tdwi.org.

BPM Architecture

Illustration 4. Based on 360 respondents.

0 10 20 30 40 50

50%

25%

21%

14%

11%

7%

6%

2%

Data warehouse

Independent data mart

Dependent data mart

Built into operational application

Federated architectureDesktop database

or fileNot sure

Other

www.microstrategy.com

13VOLUME 2

CONSOLIDATING ANALYTIC SILOS

he one word that best describes the state of analytic data in large organizationsis “fragmented.” Despite their best intentions, CIOs are struggling to deliverconsistent data that provides a single view across the enterprise. CIOs who seek

this so-called “single version of the truth” must feel like they are playing an endlessgame of whack-a-mole—every time they stamp out a renegade analytic silo, anotherpops up elsewhere.

Consequences of Proliferation. The problems that result from this rampant prolifera-tion are twofold. First, executives become frustrated because they can’t get the datathey need to assess the performance of their company. They get apoplectic when man-agers spend more time in meetings arguing about whose data is right rather thandeveloping strategies and plans to achieve corporate goals. Without a consistent viewof performance across the enterprise, executives know they can’t do a good job run-ning the company.

Second, these redundant, non-integrated analytic structures are wasteful. Whendeployed as physically distinct systems with separate hardware, storage, softwarelicenses, data feeds, and staff, these systems can drive up data warehousing costs by 30to 50 percent, according to some data warehousing managers. Consolidating thesestructures can save organizations millions of dollars a year and deliver a quick returnon investment (ROI).

Remedies for ProliferationThe Disease Is the Cure. Ironically, the cure for the proliferation of analytic silos is tocreate another analytic structure. To stamp out analytic silos, organizations need toimplement an enterprise data warehouse (EDW) that provides sustenance for all past,present, and future analytical structures. The key is to plant the EDW deeply enoughand broadly enough within the information architecture and corporate culture that itbecomes the de facto analytic structure within the organization.

In Search of a Single Version of Truth

T

From the TDWI Research Report In Search of a Single Version of Truth:Strategies for Consolidating Analytic Silos, sponsored by AscentialSoftware Corporation, Hummingbird Ltd., Informatica Corporation,Teradata, a division of NCR, and Unisys Corporation. This report was published in August 2004.

by Wayne Eckerson

CONSOLIDATING ANALYTIC SILOS

14 TDWI’S BEST OF BUSINESS INTELLIGENCE

The EDW only works, however, if the organization comestogether to hammer out definitions and rules for common-ly used terms and calculations, such as “net margin” or “sale”or “profit.” Standardizing the meaning of shared data ele-ments—often referred to as metadata—is often more chal-lenging than consolidating the actual physical structures.Once this exercise is completed, the EDW becomes therepository for shared data, rules, definitions, and othermetadata used by multiple analytic applications.

Trends in Analytic ConsolidationA Big Job. Consolidating duplicate data and redundant ana-lytic structures is not for the faint of heart. That’s becausemost organizations have dozens of analytic structures thatthey would like to consolidate. Organizations have an aver-age of two data warehouses, six independent data marts, 4.5 ODSs, and 28.5 spreadmarts left to consolidate (seeIllustration 1). To date, they’ve only consolidated about one-third of all the structures in their environment. But acquisi-tions, mergers, and reorganizations make this number a per-petually moving target.

On average, organizations have consolidated a slightly high-er percentage of data warehouses (42 percent) and a slight-ly lower percentage of spreadmarts (22 percent). This makessense given the relative numbers of these structures. In fact,it’s safe to say that the smaller the structure, the more like-ly it is to proliferate and the harder it is to consolidate.“Every week during our assessment phase, I discovered anew spreadsheet that was a source for customers or con-tacts. In the end, I found more than 23 files in spreadsheets,FoxPro, ACT!, GoldMine, and Paradox databases that con-tained customer information,” says Wanda Black, directorof information resource management at a privately heldmanufacturing firm.

Project DurationTime to Consolidate. Not surprisingly, the most complex ana-lytic structures take the most time to consolidate and thesimplest structures take the least time. Our survey showsthat data warehouses take an average of 9.75 months toconsolidate (the average of planned and completed consol-idations), followed by operational data stores at 6.82months, data marts at 6.02 months, and spreadmarts at5.69 months. (See Illustration 2.)

Twenty-Three Years of Hard Labor. If we multiply the num-ber of analytic structures that organizations plan to consol-idate by the average time to consolidate each type of struc-ture, then it will take organizations 277 months—or 23years—to complete their projects if done sequentially!Obviously, organizations need to consolidate multiplestructures in parallel—or implement a true EDW that canreplace all warehouses, marts, and spreadmarts at once, ormore likely, over a period of several years.

Cost JustificationAlthough reducing costs is an important reason to consoli-date analytic silos, it usually takes a backseat to more strate-gic concerns. “We did some ROI analysis, but that’s not thedriver. Although we expect to reduce costs by 30 percent ormore, what is really driving this project is the need to com-ply with the Basel Accord and improve the bank’s risk analy-sis capabilities,” says the lead architect at an Australian bank.

According to our survey, less than a third of organizationsanalyze what it costs to support independent data marts ordata warehouses or migrate them to a new environment.Similarly, less than one-third calculate the ROI of migratinganalytic structures to a new environment.

Illustration 1. Organizations plan to consolidate all types of analyticstructures, especially central warehouses and independent marts.Based on 615 respondents.

1.6

2.9

2.3

8.0

2.1

6.0

4.6

28.5

Data warehouses

Independent data marts

Operational data stores

Spreadmarts

Average Number

Already Consolidated Plan to Consolidate

Completed versus Planned Consolidations

Illustration 2. Data warehouses take the longest time to consolidate,approximately 9.75 months. Based on 209 respondents who havecompleted a consolidation project.

9.69

5.65

6.62

4.44

9.8

6.4

7.01

6.93

Data warehouses

Independent data marts

Operational data stores

Spreadmarts

Months

Completed Planned

Time to Consolidate

CONSOLIDATING ANALYTIC SILOS

15VOLUME 2

Average Costs and Projected ROI. However, TDWI researchshows that consolidation projects deliver a hefty payback onthe investment. On average, it costs $614,000 a year tomaintain an independent data mart and $544,000 to migrateit. It costs $1.597 million a year to maintain a data ware-house and $1.171 million to migrate it. (See Illustration 3.)

Among organizations that calculated ROI for their projects,the average ROI was $3.34 million over 2.1 years. (See Illus-tration 4.) Our estimates match those generated by AlstomPower, which is now building an EDW to consolidatenumerous analytic structures, including a data warehouse, areporting repository, two departmental data marts, andnumerous Access databases. Alstom expects a $3.5 millionpayback over 2.8 years from its project, according to MichaelSykes, U.S. manager of data warehousing at the company.

The most promising candidates for consolidation are datamarts that run on different platforms and are managed byseparate IT staffs. Consolidating these independent data

marts can save companies more money than those that runon the same platform and are managed by the same IT staff.Of course, as we have seen, saving money is not the primeincentive for consolidating marts; the key driver is deliver-ing consistent information to the enterprise.

Architectural Approaches to ConsolidationEnd-State ArchitecturesOnce an organization has made a commitment to consoli-dating analytic structures, it needs to plan how to migratefrom its current chaotic environment to a standardizedarchitecture. Generally, once an organization has livedthrough the pain of analytic silos, it is adamant about cen-tralizing as much data, resources, and infrastructure as itpossibly can. “A centralized environment ensures that peo-ple are looking at the same source of information and get-ting consistent reporting and interpretations,” says RussVaughn, formerly a data warehousing manager at Bank ofAmerica. “Once something leaves our hands, we can’t guar-antee the validity of the information.”

Eight Migration Options. Through interviews with dozens oforganizations that have consolidated analytic silos, we havedefined eight migration strategies. Some organizations onlyuse one strategy; others adopt different strategies at variousphases in the migration to a consolidated environment; andothers are forced to switch strategies as business eventschange. The eight consolidation strategies in a nutshell are:

Physical Strategies 1. Rehost. Move existing analytic structures onto a

single platform.

Centralized Strategies 2. Start from Scratch. Build a new data warehouse instead of

designating or merging existing ones.

3. Designate and Evolve. Designate an existing data ware-house or mart as the corporate standard and migrateother structures to it, either immediately or over time.

4. Backfill. Implement a staging area/warehouse behind exist-ing data marts to consolidate extracts and data for marts topull from.

5. Synchronize. Synchronize remote operational data storesfrom a central reference repository.

Distributed Strategies6. Conform Data Marts. Conform the data models of exist-

ing data marts by standardizing shared dimensions.

Illustration 3. Less than a third of organizations calculated what it coststo support or migrate analytic structures. Based on 573 respondents.

27%

28%

35%

10%

31%

26%

43%

0%

26%

36%

38%

0%

Yes

No

Partially

Not sure

Cost to Support Cost to Migrate ROI

Did You Estimate Costs and ROI?

Illustration 4. The average ROI for consolidation projects is $3.34million. Based on 150 respondents. Forty-four percent of respondentsselected “don’t know.”

Average = $3.34 million

18%

16%

8%

5%

5%

4%

< $500K

$500K–$1M

$1M–$2.5M

$2.5M–$5M

$5M–$10M

$10M–$50M

Estimated ROI for the Consolidation

CONSOLIDATING ANALYTIC SILOS

16 TDWI’S BEST OF BUSINESS INTELLIGENCE

7. Create a Mart of Marts. Create an enterprise view acrossdata marts by extracting data from them to create a newsuperset data mart.

8. Distributed Query. Create an enterprise view by queryingmultiple marts and reconciling results on the fly.

Some consolidation strategies are best suited for differentorganizational structures or strategies. (See Table 1.) Forinstance, the “start from scratch” strategy is best used whentwo equally sized companies merge. The “designate andevolve” strategy is best used when a bigger companyacquires a smaller one. Rehosting is used by companieslooking for quick cost savings, while synchronization isgood for large companies with lots of operational applica-tions that need to share the same reference data.

The distributed strategies are good interim solutions whilethe organization implements a centralized strategy.However, the conformed mart strategy can also be deployedin a centralized fashion if all the marts are logical schemaswithin a single instance of a central database. We also dis-covered a correlation between the level of metadata integra-tion required and the speed at which the strategy can bedeployed. It’s no surprise that the strategies that are quick-est to deploy—rehosting, mart of marts, and distributedquery—involve the lowest level of metadata integration.

Conversely, those strategies requiring the highest levels ofmetadata integration—especially “start from scratch”—arethe most time consuming to deploy, often taking severalyears to complete. Ultimately, the approach an organizationuses to migrate to an environment that delivers standardized

information depends on many factors, including its culture,organizational structure, technical expertise, funding, andavailable time to accomplish the migration.

ConclusionToday, organizations are plagued by the proliferation ofanalytic silos, which make it difficult to deliver consistentinformation across the enterprise. Organizations are nowstarting to consolidate these silos into an enterprise datawarehouse (EDW) to deliver a single version of truth andreduce overhead costs.

A Long Way to Go. But organizations still have a long way to go. They’ve consolidated one-third of their analytic structures on average and have dozens more to go. Since it

takes between four and 10 months to con-solidate analytic structures, these consolida-tion projects can last years.

The good news is that the ROI for theseprojects is impressive. Organizations thatconsolidate multiple analytic structuresmake $3.34 million on their investments inabout two years. In addition, there are eightproven migration strategies that organiza-tions can use as guides. The migrationapproach an organization uses depends onmany factors, including its culture, organi-zational structure, technical expertise, fund-ing, and available time to accomplish themigration.

Critical Success Factors. The key to a suc-cessful consolidation project is to gain top

management commitment, and put together a project plan,team, and tools that help you migrate incrementally to yourtarget environment. It’s critical to assess your current envi-ronment so you can prioritize efforts and gain momentum forthe project. “I try to keep it simple,” says Russ Vaughn. “Youneed to know what steps one, two, and three are but you can’tfocus on step two until you finish step one. I stay on a 60-daycommitment list.” n

Wayne Eckerson is Director of Research at TDWI. He can bereached at weckerson@tdwi.org.

Table 1. Summary of characteristics of the eight consolidation strategies.

STRATEGY Organizational Structure Speed to Deploy Metadata Integration Hardware Savings

1. Rehost Centralized—Fast Immediate High HighFocus on quick cost savings

2. Start from Centralized— Slow High High—Scratch Merger of equals Eventually

3. Designate Centralized— Medium Moderate High—& Evolve Merger of unequals Eventually

4. Backfill Decentralized—But central Medium Moderate Extra costsIT sells shared infrastructure for DW

5. Synchronize Centralized— Slow Moderate Extra costs Master data management for hub

6. Conformed Decentralized—But with Medium Moderate NoneData Marts enterprise view

7. Create a Decentralized—But top Fast Minimal Extra costs Mart of Marts execs need single view for data mart

8. Distributed Decentralized – Quick fix Fast Minimal NoneQuery

17VOLUME 2

CRM/ERP DATA

Growing Marketplace. Following a hiatus after the year 2000, the use of appli-cation packages is once again growing rapidly across all business areas, espe-cially in front-office CRM systems and back-office ERP systems. This growth

is occurring not only for application packages that handle business transaction (BTx)processing, but also for packaged solutions that support business intelligence (BI) anddata warehousing (DW).

The Application Suite. Several vendors offer complete suites of products that contain aset of integrated applications that support both BTx and BI processing for a widerange of business areas from the front office to the back office.

A Complete IT Infrastructure. Application package suites frequently come with devel-opment tools for building BTx and BI applications, middleware for integrating otherapplications and data, and an application server environment for running and man-aging packaged and custom-built applications.

Using Business Intelligence to Drive the Business. In parallel with the growth in the useof BTx and BI application packages, the BI industry is also going through a period ofsignificant change. BI is no longer used just for doing strategic and tactical reportingand analysis, but also for driving and optimizing daily business processes and work-flows. BI is no longer nice to have, but essential to business success. The rapid evolu-tion of application packages, application package suites, and business intelligence rais-es several important questions.

1. In what business areas are companies deploying BTx and BI application packages today?How successful are application suites, what benefits do they offer, and what strategiesare being used by organizations to deploy them throughout the enterprise?

Developing a BI Strategy forCRM/ERP Data

A

From the TDWI Research Report Developing a BI Strategy for CRM/ERPData, sponsored by Hummingbird Ltd., Hyperion Solutions Corporation,Informatica Corporation, SAP America, Inc., Siebel Systems, Inc., andTeradata, a division of NCR. This report was published in October 2004.

by Colin White

CRM/ERP DATA

18 TDWI’S BEST OF BUSINESS INTELLIGENCE

2. How does the use of packaged application solutions in business units impact existing and future BI projects? Whatstrategies should be used to integrate data between appli-cation packages and with an existing data warehousing environment? How does the use of application packagesand application suites affect BI/DW product selection?Should companies focus on buying their solutions from a single applications vendor, or should they buy best-of-breed products?

3. How do evolving BI/DW technologies such as in-line andreal-time BI, performance management, predictive analysis,and support for XML and Web services affect the selectionand use of packaged application solutions?

The objective of this report is to answer these questions andoffer suggestions for addressing the issues they raise. The goalof the report is also to suggest possible strategies for buildinga BI system and underlying data warehouse for processingCRM and ERP data managed by application packages.

The Landscape for PackagedApplication SolutionsThis report compares and contrasts different approaches todeveloping business intelligence applications using transac-tion data managed by application packages from vendorssuch as Oracle, PeopleSoft, SAP, and Siebel. To put thistopic into perspective, this section of the report discusseshow business users run, optimize, and communicate aboutbusiness processes. It then looks at how application pack-ages have evolved to support those processes.

Business Transaction ProcessesAt the heart of any corporate data processing system are theBTx processes for handling day-to-day business operationssuch as front-office customer relationship management(CRM) middle-office finance and human resources, andback-office enterprise resource planning (ERP) The tech-niques and tools for developing and deploying these BTxprocesses and their underlying applications have evolved overseveral generations of computer hardware and software.

Custom Coding. Initially, BTx applications were customdeveloped using programming languages such as Assembler,COBOL, and PL/I. Over time these languages were super-seded by object-oriented languages such as C++ and Java.

BTx Application Packages. As BTx application usage grew,development groups looked for faster ways of buildingapplications. This need led to visual programming tools andpackaged BTx application solutions. Initially, packaged

BTx applications provided stand-alone solutions for specif-ic business areas. More recently, however, application ven-dors have been marketing complete suites of integratedpackages that meet customer needs for deploying a widerange of related and interconnected BTx applications.

BTx Application Package Evolution. The use of BTx applica-tion packages increased as companies struggled to replaceand reengineer their aging legacy systems in anticipation ofyear-2000 issues. This was especially the case in ERPdeployments. At the beginning of 2000, the pace of migra-tion to packaged solutions slowed. Both ERP and CRMpackages suffered a number of well-publicized failures. Asboth types of BTx packages have matured, however, the rateof adoption has once again increased. The move to reduceIT costs and standardize on products has also encouragedcontinued growth in the use of BTx application packages.

Using BTx Application Package Suites. BTx applicationpackage suites are used by organizations to replace in-houselegacy applications and also stand-alone application pack-ages. The pros and cons of employing an application pack-age suite are well documented. Their main benefits are busi-ness models based on best industry practices, faster deploy-ment time, reduced IT resources, and greater integrationacross applications.

When selecting an application package suite, you shouldconsider issues like reliance on a single vendor, and the cus-tomization and maintenance effort required to modify thesuite to match the organization’s business processes. Thecost of application suites is another common issue. Giventhe objective of a suite is to reduce development and main-tenance costs, then the total cost of ownership (TCO) is abetter cost comparison against other approaches.

Several companies interviewed for this report said their sen-ior management had decided to adopt a single vendor suiteof BTx packaged solutions because they wanted to reduceIT expenditure. The level of investment organizations hadmade in a suite was the driving force behind this decision.Executives wanted to offset the cost of deploying a suitewith a corresponding decrease in IT resources and softwareoverhead. Moving from multiple dispersed applicationpackages to an integrated product set from a single vendorhas obvious financial benefits, but this strategy will taketime to implement and has key implications for existingand future BI projects.

Many Organizations Have Deployed Multiple BTx Packages.We asked respondents in our survey about the number of vendors who supply BTx application packages to their organizations: 29 percent are using packages from

one vendor, 26 percent from two vendors, 16 percent fromthree vendors, and 5 percent from four vendors (seeIllustration 1). Some 16 percent are using packages fromfive or more vendors. In total, more than 50 different pack-ages are listed in survey responses. It is likely that somerespondents answered this question for their own divisionor business unit, and in reality the number of packages usedby organizations may be higher than the numbers shown.

Business unit autonomy and company mergers and acquisi-tions are two reasons why companies have multiple pack-ages. Also, individual business units have deployed applica-tion solutions at different points in the BTx applicationpackage lifecycle, and the maturity of packaged solutionsfor particular business areas often determined the develop-ment approach and products used.

Applications Areas Supported by BTx Packages. The mainbusiness areas supported by BTx application packages (see

Illustration 2) include finance (78 percent of respondents),sales (54 percent), and human resources (49 percent). It wasvery common for organizations to support the front officeand back office with CRM and ERP packages from differ-ent vendors. Also, middle-office human resource andfinance departments frequently have their own stand-alonepackaged BTx solutions.

When comparing the number of packages used with corpo-rate revenue, 50 percent of companies with three or morepackages have revenues in excess of $1 billion. Surprisingly,19 percent of companies with three or more packages haverevenues of less than $100 million, which shows that evensmaller companies have to deal with the issue of handlingmultiple packages.

Business Intelligence ProcessesBI application development mirrors the evolution of BTxapplications. Organizations began initially by coding theirown BI applications, but rapidly progressed into the use ofinteractive BI productivity tools and packaged BI applications.

BI Application Package Use Is Growing. Like the BTx appli-cations market, the BI area is also seeing growth in the useof application packages. The worldwide BI applicationpackage market is forecast to reach over $4.8 billion in2007, according to IDC. All three of the IDC BI applica-tion package market sectors are projected to enjoy revenuegrowth through 2007. CRM analytics will grow the fastestwith a compound annual growth rate of 12.9 percent, fol-lowed by financial analytics/performance management at10.3 percent, and operations/production analytics at 7.4percent. Financial analytics/performance management,however, is by far the largest market segment at present.

Survey results show (see Illustration 3) that 30 percent ofcompanies are not using BI application packages from theirBTx application vendors, while 37 percent are using justone package. The remaining 33 percent are using two ormore of these solutions. Leading business areas beingaddressed by BI packages (see Illustration 4) include finance

Illustration 2. Which business areas do your BTx application packages support? Based on 522 respondents.

Illustration 1. How many software vendors supply your organizationwith BTx application packages? Based on 542 respondents.

CRM/ERP DATA

19VOLUME 2

Number of BTx Application Package Vendors in Use5+ vendors

16%

4 vendors5%

3 vendors16%

2 vendors26%

1 vendor29%

0 vendors8%

78%Finance/accounting

54%Sales

49%Human resources

45%Customer service

42%Order management

34%Procurement

34%Marketing

31%Supply chain management

22%Manufacturing

12%Other

Business Areas Addressed by BTx Application Packages

Illustration 3. How many BTx application vendors supply your organ-ization with BI application packages? Based on 545 respondents.

Number of BI Application Package Vendors in Use

5%3%

6%19%

37%30%

5+ vendors

4 vendors

3 vendors

2 vendors

1 vendors

0 vendors

CRM/ERP DATA

20 TDWI’S BEST OF BUSINESS INTELLIGENCE

(70 percent of respondents), sales (56 percent), and mar-keting (45 percent). The top two business areas,finance/accounting and sales, are the same as those for BTxapplication packages. Marketing, however, is more domi-nant in BI processing than it is in BTx processing.

BI Strategies for BTx Package DataHaving outlined how companies use BTx application pack-ages, we now move on to look at different BI strategies forreporting on and analyzing the data from those packages,and for solving the data integration issues involved.

Possible BI/DW StrategiesThere are three main BI/DW approaches for processing datamanaged by BTx application packages. These can be thoughtof as the Adopt, Avoid, and Accommodate strategies.

Some of the main business and technology requirementsthat affect product choice include political issues, existingBTx and BI environments, IT skills and resources, productfeatures and functions, integration capabilities, perform-ance and scalability, total cost of ownership (TCO), andvendor relationships and support.

What BI/DW Strategies are Companies Actually Using?The most popular BI/DW strategy for handling BTx package data according to TDWI survey results is Accommodate, followed closely by Avoid. Of the three strate-gies, 32 percent of respondents rated the Accommodate strat-egy as being a “very high” or “high” match to their corporateBI/DW approach. The Avoid strategy matched 29 percent ofcorporate BI/DW approaches, while the equivalent figurefor the Adopt strategy was 24 percent (see Illustration 5).

Another way of looking at these results is 56 percent of respondents are using BI/DW solutions from BTx applications vendors (Adopt or Accommodate strategies),

and 61 percent are using third-party BI/DW products(Accommodate or Avoid strategies). These figures clearlyshow the direction of companies toward incorporatingBI/DW solutions from BTx applications vendors in theirBI environments.

Although 29 percent of respondents rated the Avoid strategyas being either a “very high” or “high” match, nearly half ofthe same companies have purchased BI/DW products fromtheir BTx application package vendor. One explanation forthis anomaly is that individual business units may have a dif-ferent BI/DW strategy from that of the central IT group. Abusiness unit may implement a BI/DW solution from a BTxapplication package vendor, even though the central ITgroup may have adopted an Avoid strategy.

Other Factors That Influence Product Choice. The survey alsolooked at some of the factors that influence product choice.Two key factors are integrating best-of-breed products forgaining maximum functionality, and the total cost of own-ership (TCO) of the BI/DW application solution. A best-of-breed approach is used by 62 percent of respondents,which is close to the figure of 61 percent for people employ-ing the Avoid or Accommodate approaches (i.e., people usingthird-party BI/DW products). Nearly half of the respon-dents rate TCO as “very high” or “high” in importance.TCO requirements should be independent of strategy.

ConclusionThe TDWI survey and follow-up interviews show thatmost companies have a complex mixture of applicationsand data stores for handling CRM and ERP data, and forreporting on and analyzing this data for corporate decisionmaking. Providing executives and managers with a consis-tent and integrated view of business operations in such anenvironment is a difficult task, and companies use a varietyof BI and DW techniques and products to support this

Illustration 4. Which business areas do your BI application packages(from BTx application vendors) support? Based on 482 respondents.

70%Finance/accounting

56%Sales

45%Marketing

37%Customer service

34%Order management

31%Human resources

26%Procurement

23%Supply chain management

17%Manufacturing

11%Other

Business Areas Addressed by BI Application Packages

% of respondentsVery High High Medium

0 5 10 15 20 25 30

Illustration 5. To what degree do these statements match yourBI/DW strategy for processing BTx package data?

We selectively implement BI/DW solutions from packaged BTx ven-

dors (Accommodate strategy)

We avoid BI/DW solutions from packaged BTx vendors

(Avoid strategy)

We standardize on BI/DW solu-tions from packaged BTx vendors

(Adopt strategy)

BI/DW Strategies for Processing BTx Package Data

CRM/ERP DATA

21VOLUME 2

task. However, some common patterns and trends can beseen in the survey findings.

1. The deployment of BTx and BI application packages andsuites is growing, especially in business areas such asfinance, sales and marketing, and human resources.

2. Many companies are content to allow individual businessunits to deploy their own BTx and BI application packages.These packages offer many advantages to the individualbusiness units, but they can complicate the creation of anenterprise data warehouse and cross-business-unit BI pro-cessing. Data integration in this environment is often done bycopying warehouse data between a business unit applica-tion package data warehouse and a central enterprise DW.

3. To increase the ROI of their IT software investments, somesenior executives are mandating the use of an integrated BTxand BI product set from a single vendor across all businessunits. Whereas this approach helps eliminate the data ware-house and BI issues outlined in item 2 above, it is unlikely inthe short term that these companies can eliminate the use oflegacy applications and individual application packagescompletely across all business units.

4. Although the creation of an enterprise DW is still the goal, forthe foreseeable future many companies will need to employ ahybrid DW architecture that can accommodate BI tools andapplication packages from both BTx application and third-party vendors. Such an environment requires an open BI/DWframework that can easily absorb new BI/DW products andtechnologies as the marketplace evolves.

5. Key requirements for BI/DW for application packages are anopen and integrated framework for handling data and meta-data, scalability and performance, total cost of ownership,ease of use, and analysis and reporting power.

6. Many companies are nominating business process ownersin each business unit to improve communications with ITabout the business use of application packages, and toidentify business and information requirements for new BI applications.

7. Business intelligence is now being used not only for strate-gic and tactical decision making, but also to drive day-to-day business operations. In some cases, this is beingachieved by integrating BI processes into the businesstransaction workflow.

8. Many companies are focused on reducing IT costs, ratherthan using modern BI technology to build smarter busi-nesses. This is making the creation of an enterprise datawarehouse for cross-business-unit planning and action difficult to create. IT organizations need to work closelywith senior executives to explain the business benefits of integrating business intelligence from the demand side ofthe organization, with that of finance and the supply side of the organization. n

Colin White is President of BI Research. With more than 34 years of IT experience, he has been a consultant fordozens of companies around the world, and is a frequentspeaker at leading IT events. For more information, write to info@bi-research.com.

22 TDWI’S BEST OF BUSINESS INTELLIGENCE

DATA INTEGRATION

hen a prospective clientrecently asked my opinionabout his company building

an enterprise data warehouse, Ithought about Yogi Berra’s line: “Noone goes there these days. It’s toocrowded.” After all, doesn’t everyonehave a data warehouse?

These days the presumption that inte-grating data requires a centralized plat-form is an increasingly debatable one. AtMartin Marietta in the 1980s, my col-leagues and I were awestruck at what apractical and downright elegant solutionour data warehouse represented. In addi-tion to the benefits of mainframe offloadand direct end-user access, the datawarehouse enabled companies to focuson the information integration andanalysis needs of the business users.Furthermore, our enterprise data ware-house imposed the rigor to migrate datain a structured and systematic way.

Distributed Data WorkaroundThe success of centralized databasesenabled many organizations to provide a“single version of the truth” through themigration and copying of data fromoperational systems to the data ware-house. The challenge here was the neces-sary rigor required to integrate new data

sources. While the time investment wasworthwhile for new (and ongoing)busi-ness needs, the investment seemed ques-tionable for intermittent or ad hoc datarequirements.

At Martin Marietta, we often pulleddata from multiple systems to supportone-time-only reports because the datato support these reports didn’t exist on asingle platform. While the whole pur-pose of database technology is to simpli-fy data access and support analysis, wewere spending more time loading datathan analyzing it. For this particular sit-uation, distributed database architec-ture, while immature, was a reasonablesolution. Performance wasn’t stellar, butwe could bring data together in a fash-ion that supported one-off needs with-out migrating tables and reinventingapplications. In those days, the networkbandwidth, storage, and processingpower necessary to migrate large datacontent was more expensive, and find-ing a spare hundred gigs for a week wassignificant.

Since then, data volumes have increaseda thousandfold or more. Unfortunatelyfor the distributed database architecture,bandwidth didn’t evolve as quickly, andmost IT departments abandoned dis-

tributed databases for large-scale analy-sis. Clustered SMP, massively parallelprocessing, middleware, and other buzz-terms have inspired hope for new waysto integrate data, but they’ve also cloud-ed perceptions of what’s practical.

One way of looking at data integrationis to identify the types of business ques-tions that require integrated data, butnot in massive quantities. I’ve classifiedthese questions into two categories:

1. Detailed Inquiry.These are questions focused ondetails about a single business or sub-ject area. For instance, an end user inFinance needs specific financial met-rics from multiple business units.

2. Content-Bound Analytics.For instance, to review a customer’scomplaint, a support center represen-tative doesn’t need access to the entirecustomer base and all of their associ-ated purchase histories, just access tothe individual’s account informationand purchase details.

In each of these circumstances, thebusiness question is predefined via anapplication. The application or soft-ware package generates a discrete and

Architectural Alternatives for Data Integration

W

Part I of this article appeared in TDWI FlashPoint e-newsletter July 2, 2004;Part II appeared on September 22, 2004. FlashPoint is a TDWI MemberBenefit. For more information about TDWI Membership, contact membership@tdwi.org.

by Evan Levy

Part I

23VOLUME 2

DATA INTEGRATION

usually repeatable query and needs avery specific set of data in return. Thesetypes of questions don’t require the so-called “full table scan”—or any scan atall. They simply require an underlyingknowledge of where that data is andwhere to go get it. Usually, the answerset is no more than a single record.

New Choices Companies can now be more clever andmore deliberate about integrating databased on our business and usage require-ments. Today’s database-oriented opti-mizer and metadata technologies canidentify which data to move and findthat data at its operational source.Integration can be handled using simple(but well-defined), source-orientedmetadata. The challenge has been sim-plified: rather than worrying about find-ing the source data and moving it with-in a constrained overnight window, thisnew class of integration methods forcesthe staff to merely identify the datasource. The tool can then determine theappropriate navigation and selection cri-teria to return the appropriate records.

ERP systems have become a de factotechnology component of many compa-nies to support a single point for allfinancial and resource information for acompany. Many companies are so capti-vated by these packages that they arewilling to modify their operationalmethods to match the way the actualproducts function. Good or bad, thiscapability has introduced a new era ininformation integration. Previously het-erogeneous systems like HR, generalledger, and inventory systems now coex-ist in ERP environments, alleviating theneed for the tremendous amount of cus-tom development that used to be neces-sary to integrate operational data. Nowyou can just buy an ERP suite.

EAI (enterprise application integration)tools have come into existence as ameans of simplifying data navigationand extraction from packaged applica-tions. After all, while ERP systems inte-

grate data for their own use, sharing theirdata with other applications is a differentissue altogether. From a data warehous-ing perspective, EAI technology can dra-matically simplify data extraction andmovement between the data warehouseand different systems. Leading ETL ven-dors now leverage EAI technology intheir toolsets.

Similarly, EII (enterprise informationintegration) has emerged to supportdata integration between multiple, dis-parate systems. The idea is to packageprebuilt data navigation and extractiontechnology (such as EAI) to enablemetadata-driven data integration. Thedata is actually integrated with little orno code development.

EII addresses questions such as: Whyload a billion customer records when I’llonly need to access a single customer?And why integrate all my data sources ifI only need a specific view of informa-tion from two or three individualsources? As the two categories I identi-fied—detailed inquiry and content-bounded analytics—suggest: Why find,gather, extract, cleanse, and load hugequantities of detailed data when mybusiness requirements tell me that all Ineed are several discrete data elements ata given point in time?

Real-World Data IntegrationThe advent of strategic programs likebusiness performance management andcustomer relationship management havefurther defined the need for fast analysisof discrete operational data. A singlestrategic project won’t justify an enter-prise data warehouse, yet companiesseeking to get closer to business opera-tions or retain more customers can’t livewithout analytics.

Consider a banking example: A cus-tomer finds out she has overdrawn herchecking account and contacts thebank’s call center. While an enterpriseview of the customer is helpful (such asthe customer’s mortgage balance, line of

credit details, or even her value score), itwon’t help the CSR address this particu-lar problem. Access to the customer’saccount number and most recent trans-action is what’s needed.

There’s also an issue regarding datalatency. Most data warehouse contentfreshness is delayed due to the nightly orweekly processing window. Enterpriseintegration only works when therequired data is ready and available—inthis instance, only operational systemswill do. EII can provide SQL databaseaccess while conforming the navigation-al restrictions of the operational system.This is feasible because the applicationonly requests the records necessary, andintegrates them outside the operationalsystem.

So, ERP, data warehouse, ODS, EAI,EII… How do we know when to usewhat technology? It has everything to dowith your business requirements. Notevery data integration problem requiresmassive quantities of data. Your toolboxshould be able to address any integrationproblem (regardless of size) withoutalways requiring big servers or complexsoftware. Your requirements will driveanswers to the following three questions:

• Do business users need discrete data elements?

• Do they need an integrated view of the data from multiple sources?

• Do they need an aggregated answer set?

So where does this leave the data ware-house? The data warehouse will remainthe workhorse for ad hoc queries againstcross-functional data and to support thesingle version of the truth across theenterprise. Data warehouses are stillmandatory for doing large-scale analysisof historical detail and providing thecoveted single version of the truth.

For now, anyway.

24 TDWI’S BEST OF BUSINESS INTELLIGENCE

DATA INTEGRATION

knew things were a bit unusualwhen Neil Raden told me they’dadded a third sign-up sheet to my

TDWI Night School in San Diego. Andthe sheet, said Neil, was filling up. I’dtaught Night School before and I’dalways had good attendance, but whenthe sign-up sheet hit 60, I figured thetopic of integration alternatives to datawarehousing had some traction.

And I was right. Not only did we havea large crowd, but I was unable toanswer everyone’s questions in theallotted 75 minutes. Given that forevery person who asks a question thereare several people thinking of askingthat same question, I thought I’d covera few of the topics that seemed to betop-of-mind with Night School atten-dees, and share the answers with theFlashPoint audience.

Question: “You spent almost as muchtime covering ERP as a data integra-tion alternative as you did the otheralternatives you discussed. Why didyou give ERP so much airtime?”

We find that there is a lot of “noise” outthere about ERP (enterprise resourceplanning) and a lot of CIOs who arelooking for reasons to use ERP systemsas a single behemoth platform for dis-parate applications. The fact is, everyERP vendor positions its product offer-ings as a solution for data integration.

Many people forget that when they havean ERP system, they’ve already integrat-ed the data from the subject areas neces-

sary to operate the business. The wholepremise of ERP is that the data is inte-grated: I can’t sell or deliver a product ifit’s not already in inventory.

ERP packages do store and generate alot of heterogeneous information.Sales, accounting, and inventory dataalready coexist, so—depending onyour business requirements—using anERP system as an integration platformmay be the shortest distance betweentwo points.

Plus, practically speaking, we’ve seenthat data integrated through ERP hasmore business exposure. It’s easier forSales to monitor a customer’s paymentstatus before pitching a new product, orto get Accounting to look at productsales and supplier deliveries. When mostor all of the data that’s necessary for ana-lytics is already in an ERP system, usingthat system as the point of integration isa viable option.

Also, the ERP vendors are serious aboutintegration. Just look at SAP’sNetWeaver product. It’s become theintegration engine not only betweenSAP applications, such as MySAP andR/3, but between SAP and other sys-tems. NetWeaver’s architect likened hisproduct to “a rainbow that connectseverything.” So we clearly have to takeERP seriously as an integration option.

Question: “Our company is currentlyresearching enterprise informationintegration tools (EII). From how youdescribed EII, it could replace our

ODS and even some of what we’redoing with our data warehouse.Should it?”

There are circumstances where an EIItool can indeed address operationalreporting. But I’d encourage you to pro-ceed cautiously. EII is most successful inintegrating data from disparate sourceswhere the target data is in a singlerecord. An easy way to look at it is if Ineed to identify a particular customer’sinformation in the finance system andthe sales system, EII would be a reason-able alternative.

However, when the work is focused onintegrating all customers from these twosystems—say, generating a list of cus-tomers who haven’t paid their bills in thelast 45 days—that’s not the best use ofEII technology. Steer away from EIIwhen the goal is aggregating large vol-umes of information.

So the real answer to your question is: Itdepends on your business requirements.

Question: “I’ve heard that EAI and EIIcan be used to move data betweensystems. Do you have any simple met-rics for when each of these technolo-gies makes sense?”

Remember that enterprise applicationintegration (EAI) was meant to movedata between systems as a means formanaging business processes. Forinstance, the shipping system shouldn’tship a product until the customer’s cred-it card has cleared the purchase. So EAI

BI without a Warehouse?

IPart II

25VOLUME 2

DATA INTEGRATION

is an effective technology for moving therelevant data between applications. EAIis also useful when I need to extract datafrom a system at a particular point in its processing.

EII, on the other hand, can query andmerge data from different systems. Sothis would be useful when I need toextract data in small volumes from mul-tiple systems and pull it together. It’s notmeant for complex queries that can real-ly tax a database.

Not to confuse the issue, but it’s com-mon for EII tools to leverage EAI tech-nology under the covers. So they reallyare two different solutions to two differ-ent problems.

Question: “Can EAI and EII replace ETL tools?”

Neither of these technologies is appro-priate for the bulk processing that toolslike Informatica, Ascential, or evenCOBOL address. There is certainly anopportunity to evaluate these technolo-gies for requirements like trickle datafeeds or individual record migration.

But for companies that need real-timedata quickly, leveraging EII againstyour operational systems lets youbypass all the complexity and cost ofimplementing a near-real-time ETL sys-tem. You’ll be hitting the data directlyfrom its source.

The decision is typically made based onyour company’s existing technologyinfrastructure and the organization’s abil-ity to adopt a new tool versus developinganother from-scratch piece of code.

Question: “Deploying some of thealternatives you’ve suggested coulddrastically change the way we doreporting, and even transform our datawarehouse architecture, couldn’t it?”

Yes, it could. I frequently find that—due to the complexity of operational systems—data warehouse architecturesthat have been built to support opera-tional reporting have become a muchmore expensive proposition than someof these newer technologies.

Operational reports have migrated tothe data warehouse because, up untilnow, it was too complex and time con-suming for the source systems owners todevelop or support such reports. It turnsout that many data warehouses increasetheir number of data sources solely tosupport the specific reporting needs ofthose source systems (e.g., the proverbial“virtual data mart”). There may be anODS functioning as a “dumpingground” for operational reporting, withall the accompanying processing, stor-age, and maintenance costs attached.

While no panacea, EII can provideaccess to operational systems whilemaintaining the navigational and pro-cessing rigor. Why bring all the dataover when I only need a few discreterecords for my operational reports? EIIprovides an SQL interface that can dra-matically simplify the development ofnew operational reports.

A few of my clients have struggled withthe cost of building a “real-time datawarehouse” because they saw it as theonly alternative to capture low-latencydata. Migrating large volumes of dataonto an ODS to support detailed opera-tional reports seems wasteful if you cansimply report directly against the nativeoperational system.

Someone could argue that it’s inefficientto hit an operational system to supportreporting. But the real question is: Is itmore efficient to copy the entire con-tents of an operational system onto aseparate platform to support an opera-tional report? The answer to this ques-tion will dictate whether or not youshould keep your ODS.

Question: “I read your last FlashPointarticle and it seemed to suggest thedemise of the data warehouse. Doyou believe that?”

I don’t believe data warehouses will dis-appear anytime soon. I do believe thatthese new technologies will provide alter-natives in situations where data ware-housing isn’t the best fit for data accessand reporting. Most CIOs I’ve talked tohave complained at one time or anotherabout the escalating cost of their datawarehouse. (In a recent InformationWeekstudy, 78 percent of managers surveyedcited “getting a better return on capitalinvestments” as a “top company priori-ty.”) Some have even watched the cost ofthe data warehouse rise faster than thecorresponding value.

The technologies I describe provide alower-cost alternative for low-latencydata access than a data warehouse canoften provide. Architectures and designconventions aside, isn’t the goal to opti-mize the efficiency of getting informa-tion to the business? n

Evan Levy is a Partner and co-founder ofBaseline Consulting. He can be reachedat evanlevy@baseline-consulting.com.

T H E C O R N E R S T O N E S O F B U S I N E S S I N T E L L I G E N C E

[ S E E T H E B I G P I C T U R E ]

Managing information from disparate data sources across your organization is not easy. Once you have built a data warehouse, you have a new set of challenges: providing business areas the ability to view and interact withthe data, without taxing your staff or compromising the integrity of your systems. As you decide what solutionsyou need, keep these BI Cornerstones in mind:

� Control: Look for a vendor who can offer a wide variety of security, administration and metadata services that can be used and managed across one or many servers.

� Certainty: Require a set of querying, reporting capabilities which take full advantage of the packaged applications and data systems within your BI infrastructure. Ensure alignment with your business goals through key performance indicators and scorecarding functionality.

� Execution: Ask to see real-world customer implementations that deliver information to users in the way that makes sense to their business. Personalized, on-demand, contextual and actionable.

� Experience: Expect your BI vendor to understand how to deliver visualization options, like dashboards and visual spreadsheets, while making the data sources transparent.

L O O K F O R M O R E S T O R I E S A B O U T T H E H Y P E R I O N B I C O R N E R S T O N E S .

www.hyperion.com/cornerstones

© 2005 Hyperion Solutions Corporation. All rights reserved. Hyperion and the Hyperion “H” logo are registered trademarks of Hyperion Solutions Corporation. “Business Intelligence”

is a trademark of Hyperion Solutions Corporation. All other trademarks and company names are the property of their respective owners.

Project1 2/22/05 9:36 AM Page 1

TEN MISTAKES TO AVOID

27VOLUME 2

ForewordAs data warehousing reaches maturity, companies are

increasingly focusing on improving access to the data ware-

house. While there are generally accepted best practices for

designing data warehouses, there are few such standards for

selecting and managing business intelligence (BI) tools. When

selecting BI tools, users seldom know what they want until

they see it; and IT seldom understands the true cost, design,

and maintenance implications of a BI tool until they’ve imple-

mented it.

Complicating the scene, many companies already own multiple

BI tools thanks to departmental initiatives, mergers, or acquisi-

tions. Few executives want to terminate successful BI imple-

mentations; however, many desire the benefits of standardizing

on a single BI platform: lower cost of ownership and easier

information sharing. In selecting BI tools, there are a number of

things that companies can do to ensure they select the tool that

best fits their tactical and strategic needs. Keep in mind: the

goal is not to find the “best tool” but to find the “best fit.” Every

BI tool has its sweet spot. If you deploy the tool outside its

sweet spot, you will see its weaknesses. Ultimately, the meas-

ure of success is not which tool you select, but rather, how you

use it!

Failing to Involve Users in the BI Tool SelectionAs part of a data warehouse project, you will have mul-tiple tool selections: ETL, data cleansing, relational

database, and business intelligence (BI) tools. For the mostpart, users care little about which tools IT selects on theback end. Users, however, own the BI tool. Failing toinvolve them in the selection almost guarantees mediocreadoption. Not involving them is the equivalent of lettingthe Department of Transportation select which cars wedrive—after all, any old car will get us from point A topoint B, won’t it?

With different stakeholders involved in BI tool selection, it’seasy to see why it can be particularly painful.

• IT wants the easiest tool to deploy and maintain, as well asone that fits within existing architecture standards.

• Purchasing may want to extend a relationship with an exist-ing software supplier (e.g., an ERP or RDBMS vendor) whoalso provides BI tools, rather than work with a new BI vendor.

• End users want sexy, intuitive tools that empower them toaccess their own report or to do their own analysis.

Ideally, a BI selection committee is composed of a team thatincludes all of the above stakeholders. This enables the var-ious constituents to communicate and prioritize theirrequirements. For IT, the challenge is to translate technicalrequirements into user cost/benefits. For example, users will

Ten Mistakes to Avoid When Selecting and Deploying BI ToolsFrom TDWI’s Ten Mistakes to Avoid series, Quarter 1, 2004.

by Cindi Howson

1

TEN MISTAKES TO AVOID

28 TDWI’S BEST OF BUSINESS INTELLIGENCE

not care if the architecture is ROLAP or MOLAP, but theycare if queries are slow or if data can only be updated on aweekly basis. The types of users to include in the selectioncommittee are both current users of legacy reporting systems as well as secondary information consumers.Secondary consumers are users who may not log in to alegacy reporting system but who still need the informationfor decision making.

When users are not adequately involved in the tool selec-tion, a number of things can happen depending on who hasbudgetary control and what capabilities are missing. If youare implementing a new centralized data warehouse with astandardized BI toolset system (or independent data marts),users may continue to use the alternative tools for as long aspossible. Thus IT is left supporting two data sources andusers make decisions based on multiple versions of the truth.

If the BI tool you selected is inflexible, users will dump datainto Excel or Access and create mini data marts. This alsoleads to multiple versions of the truth and increased costs. Ifthe BI tool is difficult to use, decision makers may rely ongut feel or specialized analysts to retrieve the information.For individual business units with their own IT budget, theymay select and implement a departmental BI tool. Goodluck trying to shut down a departmental solution! Userrequirements should include features (dual y-axis charts) aswell as usage scenarios (interactivity, offline usage). For cer-tain usage scenarios, the requirements may differ. For exam-ple, users may request both offline access and Web-basedaccess. It’s important to understand what functionality theywant in the two environments. Is full drill-down requiredvia the Web as well as offline? Or is it acceptable if offlineaccess has some restrictions?

Failing to Follow a Formal Selection ProcessI’ve heard some interesting approaches to tool selections:

• “I worked with the tool in a previous company, so I thoughtit would be good enough for my new company.”

• “Management saw some demos and thought the BI vendorwould be a good company to partner with.”

• “It came for free.”

Often I hear these comments when companies are forced todo a re-selection—if that’s any indication as to whetherthese approaches work. For a limited few, they might.However, given that BI tool selections can be so contentiousand that thousands or millions of dollars are at stake,

I recommend following a formal selection process com-prised of the following steps:

1. Form the selection committee composed of both users and IT

2. Define the target user groups and usage scenarios

3. Refine the business information requirements

4. Research vendor capabilities and issues

5. Develop ranked selection criteria, revising criteria throughout the process

6. Invite vendors for onsite, scripted demos

7. Match vendor capabilities with internal requirements

8. Select a short-list of vendors to deliver proof of concept

A formal process better ensures that various requirementsare considered and prioritized. This minimizes the risk ofimplementation surprises; users know what functionalitythey will and won’t be getting, and IT understands theresources required to support the new BI tool.

Even when you follow a process, you will still have somedissatisfied stakeholders. However, you will be better able tocommunicate the committee’s decisions with documentedcosts, prioritized requirements, and acceptable trade-offs.

Analysis ParalysisIt’s important to adhere to a tool selection process,but that process must be short! If you take six months

to a year to do a BI selection, the vendor capabilities willhave changed, as will members of the selection committee.You don’t need a new player joining the team at the finalhour only to question all the work you’ve done to date. Soit’s important to agree to a strict time frame at the start ofthe BI selection. If you agree to reach a decision within twomonths, you will have to limit the number of tools youevaluate in depth. Follow the project manager’s mantra oftime, scope, and resources. If you increase the number oftools you want to evaluate, you will have to increase theevaluation time.

It’s very easy to get drawn into playing with all the cooltechnology. However, the ultimate business goal of the BItool and of the data warehouse is to deliver information andbusiness value. The faster you fulfill the business goals, themore momentum and success the BI applications will have.The longer you take to do that, the more opportunities youwill miss.

Remember: the BI tool is not nearly as important as what youdo with the BI tool.

2

3

TEN MISTAKES TO AVOID

29VOLUME 2

Failing to Differentiate Users or BI ToolsUsers have different reporting and analysis needs andinformation requirements. BI tools have varying capa-

bilities. A good BI strategy will match the user types withthe tool capabilities. Increasingly, you can buy a full toolsuite from one vendor, or you may purchase particular cate-gories of tools from multiple vendors to achieve best-of-breed functionality. Can you buy one tool for all users? Well,think about your own car-buying habits: in theory you canbuy one car for a family of four. But imagine the concessionseach stakeholder will have to make. My husband wants thegas-guzzling, yet trendy Hummer, I want the Mercedessports coupe, and the kids want a dune buggy. Instead, weget a minivan—a compromise of everyone’s needs and costs.So in selecting BI tools, you may group segments of users toshare one tool that includes compromises.

Two general categories of users are report authors andreport consumers. Report authors may be IT personnel orpower users or both. These two different categories ofreport authors will have different sets of requirements.Report consumers can range from executives who onlywant to receive a report when there is a problem, to knowl-edge workers who want to analyze data to uncover problemsand opportunities. Some companies will have four sets ofBI tools for these four user segments (IT author, power userauthor, executive consumers, and knowledge workers).Others will have one BI tool that is deployed in differentways. To identify your various user types, evaluate themagainst several criteria such as:

• Job function or role

• Job level

• Internal user versus external customer or supplier

• Level of computer literacy

• Type of data they access by subject area, level of aggregation, and variability

• Frequency of access

• What they want to do with the data once they access it

• Devices and locations for accessing the data (Web, note-book, PDA, desktop)

Next, understand the various categories of BI tools. TDWI’scourse Evaluating BI Toolsets is an excellent resource to helpwith this.

• Production reporting

• Management reporting

• Online Analytical Processing (OLAP)

• Analytic Applications

• Scorecarding

• Dashboards

• Spreadsheets

Some BI tools can be deployed in ways that allow it to crossa given category. For example, some management reportingtools allow you to build basic dashboards. So you need toconsider what additional functionality you get if you select aspecialized dashboard application. Does it fulfill critical userrequirements for a particular user segment, and at what cost?

Finally, match the user segment with the tool you’ve select-ed for a given BI tool category. Look for areas in which youcan minimize the number of tools any one user will have tolearn as well as tools that can be deployed in different ways.The following table provides generic user segments withpossible BI tool categories.

Failing to Read the Fine PrintBI software license agreements require an advanceddegree—although I can’t figure out in what! Fail to

negotiate the license carefully, and you will buy more prod-uct than you need, or conversely, significantly underesti-mate your license costs. No two vendors license their prod-ucts the same way, making it difficult to get an apples-to-apples comparison of licensing costs.

Often the purchasing agreement is developed among mul-tiple people with multiple understandings of what you wantand what you will get. As the technical customer, you mayknow how you intend to deploy the BI tool. However, thepurchasing department executing the contract may not beas educated in the finer points of which modules you need,what platforms you will run on, and which databases youwill run against.

The vendor sales person knows how to sell the software, butmay not understand technical limitations that affect licens-ing. A technical sales consultant will, but these conversa-tions are not automatic. (I agree that it’s absurd for a BI

User Segment BI Tool

Senior executives Scorecards; scheduled managementreports that generate alerts

Financial analyst Spreadsheet

IT report developer Production reporting authoring tool

Business analyst OLAP

Report consumers Interactive management reports

4

5

TEN MISTAKES TO AVOID

30 TDWI’S BEST OF BUSINESS INTELLIGENCE

sales rep not to know their own product line. But to be fair,look at car salesmen—do they know how the car works?)Further, the BI software industry has high sales forceturnover, with some vendors being worse than others. Thebest situation is when the vendor provides you with a vet-eran account manager who understands the product line indepth and understands how you plan to use the BI tools.This ensures you only buy what you need, as well asimproves customer loyalty.

I have seen companies be very diligent in the RFP process.The number of users and planned architecture are well doc-umented and communicated. However, these same require-ments are not written into the license agreement, increasingthe chances that, as deployment ramps up, the companywill be non-compliant.

Pay careful attention to user-based versus server-basedlicensing. When it’s user based, does the license restrict youto named or concurrent users? Does the software automat-ically prevent you from exceeding the licensing count? WithWeb-based deployments, companies are increasingly pur-chasing server-based licenses. For these types of licenses, paycareful attention to which platforms you plan to deploy(initially as well as in the future), which databases you willaccess, CPU limitations, and whether there is a fee for hav-ing test, development, and production environments.Finally, many companies think of a “user” as someone whologs into the system.

However, if one of your requirements is the ability to dis-tribute standard reports in PDF or Excel format via e-mail,then chances are the vendor also considers these e-mailrecipients “users.”

Maintenance fees are another aspect commonly overlooked.These fees are usually a percentage of the initial agreement,but whether it’s a percentage of the discount price or the listprice is variable. Maintenance usually includes softwareupgrades. However, some vendors consider new functional-ity as new software (not an upgrade)that requires addition-al fees and negotiations. Understand this in advance andword your contract accordingly.

Overlooking Departmental InitiativesMany companies have multiple BI tools, implement-ed by individual departments, inherited through

company acquisition, or bundled with other software pack-ages. When it comes to defining requirements, cost is oneof the largest. If your company has already made heavyinvestments in departmental licenses, training, and consult-ing, these costs should not be overlooked. On the other

hand, if you’ve purchased a BI tool that has largely beenshelfware, you have to consider those licenses as sunk costs.The software may be “free,” but implementation costs,training, and hardware are not. If other BI tools have beendeployed successfully, it’s helpful to understand the factorsthat contributed to the success. For BI selection teamsfocusing on an enterprisewide solution, this can be a some-what insulting experience that requires a high dose of diplo-macy. Someone did it before you? Someone did it betterthan you? If your goal is to standardize and reduce the num-ber of BI tools, then individual departments will naturallyfeel threatened by your assessment. Their implementation isa success: their tool should be selected as a standard.Perhaps it should be. In these cases, engaging the assistanceof an objective consultant can minimize “turf” wars andimprove communication. Finding that objective consultantis not as easy, since many consulting firms specialize in par-ticular BI products.

Defining and communicating plans for existing depart-mental initiatives is essential for any dialogue with existingBI teams. When the BI selection is complete, what will hap-pen to existing initiatives? Will they:

• Freeze the current implementation?

• Shut down the departmental initiative?

• Convert the departmental BI solution to an enterprise standard?

Clearly, the first two alternatives are the most intimidatingfor departmental implementations; the latter is what theymay fight for. If you can involve any current BI experts inthe selection committee, it will help with buy-in as well aswith requirements definition.

At the same time, you want to learn from the mistakes of ear-lier BI implementations. Did they learn the hard way that youcan’t build big cubes with DOLAP tools? Did they find thatmaterialized views sped relational queries but the company hasa limited number of DBAs who know how to implementthem? Just as you want to identify the success factors for theBI tool, you want to understand the obstacles that influenceyour selection as well as your deployment plans.

Equating Ad Hoc with a Blank ScreenFor years, users have been frustrated with a lack ofinformation access. IT has been overloaded with

backlogs of requests for more reports or modified reports.Ad hoc reporting was supposed to solve these problems.Users would be empowered with the ability to create their

6 7

TEN MISTAKES TO AVOID

31VOLUME 2

own reports, without assistance from IT. IT could provideaccess to data and would never have to write another report.

Users are not very good at defining requirements. They askfor everything just in case they might one day need a par-ticular piece of data. If they don’t ask for it up front, it maytake another year for IT to provide it. IT is also not verygood at forcing users to prioritize their requirements andrightly does not want to be accused of not delivering thedata the business needs. So often with an ad hoc query envi-ronment, many IT shops give users access to every data ele-ment in the data warehouse. Whoa! The unsuspecting busi-ness user who attempts to build a query for sales by prod-uct slowly realizes that sales can be booked by order date, byinvoice date, by list price, by discount price, net of returns,and so on! The most savvy users will persevere and will getan answer, although perhaps not the one they were expect-ing and most certainly not the same answer as the next user.More cautious users will seek help from a specialist. Theend result: users say the BI tool is unfriendly and the datacan’t be trusted!

When users ask for ad hoc access, this is not what theywant. Users want ad hoc reporting tools because they wantflexibility and fast access to information. That doesn’t meanthey want to start with a blank screen that will be populat-ed from a list of thousands of potential data elements. Nordoes it mean they want to wait a year for IT to make onesimple change to a report.

When deploying ad hoc reporting tools, IT must stillauthor standard reports. (Surprise!) Alternatively, powerusers may author these reports. Either way, you must pro-vide users with a starting point: a report template that userscan easily modify. The modified report may eventuallybecome a standard report or it may remain a personalizedview. These standard reports may include prompts thatdynamically build the report: do you want European salesor U.S. sales?

In addition to providing users with starter reports, IT mustprovide a process in which they capture new and modifiedreports so that these reports can be leveraged across theorganization. As users explore previously hidden data, theywill find more effective ways of analyzing it. While regula-tory reports may be fairly static, management reports arenot. Establishing a center of excellence with experts whoperiodically and proactively review available reports is oneway to capture what users are doing. Here too, if you thinkusers will establish this process themselves, you are mistak-en. Users barely have time to access and analyze data with-out dedicating time to ensure it’s a repeatable process.

Training Users on the Tool and not the DataBI tool training is a challenge because there is no sin-gle way to deliver it. First, just as you select BI tools

according to user segments, you must also deliver differenttraining approaches to various user segments. A senior exec-utive, who looks at scorecards and receives alerts, may onlyneed a cheat-sheet with instructions on how to log into abrowser-based BI application. A power user, on the otherhand, may need several days of classroom training.

In either case, it’s important to provide both data and tooltraining. The most effective training combines both in an inte-grated course. Users learn how to use the tool with their data.However, this can be expensive and difficult to deliver andmaintain. For every subject area, BI tool, and user segment,you need different training material and methods. Vendor-provided training material must be customized, but by whom?The data experts are not necessarily capable trainers.

Another approach to BI training is to provide generic tooltraining supplemented by data discussion workshops. Thisis easier to administer and maintain; however, it may not beas effective as an integrated course.

Bottom line: no matter which approach you use, you musttrain users on both the BI tool and the data.

Failing to Promote the BI ApplicationPromoting any application is a foreign concept formost IT shops. You are building what the users asked

for—why should you promote it? Yet it’s important to real-ize that not all users requested the BI application. Probablyonly a select few have been involved in the various plan-ning, selection, development, and implementation phases.

You should begin promoting the BI application during theplanning stages. Your messaging at this early stage should bequite high level, focusing on general capabilities and subjectareas. “In 2004, we will be delivering a new sales reportingtool.” As you move closer to delivering capabilities, themessaging should be more specific, emphasizing how thenew BI tool will benefit that particular user and when. “TheBI sales reporting tool allows interactive analysis so you canidentify and better serve top customers.”

Use a variety of media to promote the BI application, againtailored for each user segment. Promotional media includes e-mail, corporate intranet, company newsletters, staff meet-ings, or road shows. Increasingly, BI vendors look for cus-tomers to act as case studies who have successfully deployedtheir tools. BI vendors may create videos describing the bene-fits and successes of the BI application. This allows the vendor

8

9

to promote their tool externally and provides the customer anexcellent way of promoting the application internally.

When developing a marketing strategy and key promotion-al messages, it’s helpful to enlist the help of your internalmarketing or communications department.

Failing to Fix the DataUsers see a graphical tool with color-coded mapsand indicators and think it’s the answer to all

their information needs. When you start delivering poor-quality data via a sexy BI tool, the BI tool will quickly getblamed for any data discrepancies. It’s difficult for users todifferentiate between the BI tool and the data sources.

This poses a difficult challenge for project teams, who areoften split into distinct groups: the data warehouse teamand the BI application team. Rarely does the BI applicationteam have the authority to correct ETL processes and aggre-gation rules.

Yet, when these processes have errors, users no longer trustthe tool. BI tools introduce additional calculations that alsomay cause users to question the data quality.

Some data problems are programmatic: the ETL or BI ruleswere incorrect. Others are process related: the businessthought certain processes worked a particular way and stillothers are simply input errors. Programmatic errors can beresolved by quality assurance (QA) procedures. When writ-ing custom code, developers frequently have code reviews inwhich other developers review each other’s program logic.The same QA process seldom happens with BI logic—trans-formations that may happen in the metadata layer, cube, orindividual report. Fail to fix any programmatic errors beforeusers see them in the BI tool, and you justifiably face a cred-ibility issue. Some development teams have a defined datavalidation phase (does the output in the BI tool match thesource system?). While data validation helps, it’s not areplacement for a strong QA process—data may match abasic report, but not another data slice or report that con-tains numerous report-level calculations. Is the error in thereport or in the ETL process? If your QA process is less thanstellar, any new BI implementation must start with a tight-ly controlled pilot. The pilot phase, then, is used to testfunctionality as well as to uncover data quality problems.Users know in advance they may get incorrect data.

For process and input errors, production users often have tosee the data—in the BI tool—before they realize there is aproblem. Assigning a data steward—in which a businessuser (rather than IT) is ultimately responsible for data qual-ity—can help in the long term. However, changing workprocesses that improve data quality rarely happens quickly.So the debate begins about whether to implement a quick-er ETL fix that gives cleaner data, or to do the harder,longer work of fixing the root cause. It depends, of course,but my vote is to do both. It is a catch-22.

If you fix it in the ETL process, the business sees no workprocess to fix. If you don’t fix it in the ETL process, the datawarehouse and BI tool may be unusable. When users dis-cover these process-related data problems, they may devisetheir own solutions to improve data quality, such as mas-saging the data in Excel or writing complex report formulasto mask the problems. Input errors are most easily resolved(e.g., a $1,000 invoice mistakenly entered as $10,000). TheBI tool can help users uncover and correct these errors byanalysis and exception highlighting. Sales were exceptional-ly higher this month than last: was it an effective promotionor, sadly, an input error?

Bottom line: with clean data, your BI tool can be successful.With bad data, any BI tool will fail. n

Cindi Howson is president of ASK and has 10 years of businessintelligence and data warehousing experience. She can bereached at cindihowson@askcindi.com.

10

TEN MISTAKES TO AVOID

32 TDWI’S BEST OF BUSINESS INTELLIGENCE

1

business intelligence and data warehousing professionals educated

focus—to support yourprofessional development needs

World ConferencesTDWI Conferences feature world-class instructors, one-on-one sessions with industry gurus, hype-free exhibits, and opportunities to network with your peers.

Onsite EducationBring top-rated instructors and high-quality education, tailored to address your particular challenges, issues, and skill levels, directly to your location.

Regional SeminarsWe take our high-quality education and top-notch instruc-tors on the road, making it cost-effective and easy for you to get the in-depth, content-rich training you need.

FastTrack ProgramFastTrack offers intensive, role-based training that will equip you with core concepts and allow you to explore what you learn with hands-on practice.

A Decade of ExcellenceBusiness Intelligence and Data Warehousing Education, Research, and Community

TDWI was formed in 1995 with a clear mission: Provide data warehousing professionals with comprehensive, unbiased education and information to help them do their jobs more effectively. The industry has seen impressive growth and change since 1995. Because business intelligence and data warehousing initiatives are mission-critical for organizations worldwide, the professionals who build and implement these programs are invaluable, and their knowledge crucial. TDWI’s only job is to meet the professional development needs of this community.

Download TDWI’s 2005 Education Catalogwww.tdwi.org/one

30,000

TDWIBrandAd3_0205.indd 1 2/18/05 7:11:58 AM

34 TDWI’S BEST OF BUSINESS INTELLIGENCE

THE CASE FOR XML/A

Some companies are tapping XML/A to expose OLAP services to Excel, PowerPoint,Word, and other unconven-tional clients

t’s been a long time coming, but itlooks like the XML/A standard is just about ready for prime time.

Hyperion, SAP, SAS, and others alreadysupport it in shipping products, andMicrosoft will deliver a native XML/A innext year’s SQL Server 2005 release. Andnot a moment too soon: some companiesare already tapping XML/A to supportpromising use-cases—such as seamless-ly exposing OLAP services in Excel,PowerPoint, Word, and other unconven-tional clients.

A number of vendors—includingHyperion Solutions Corp., SAP AG,and SAS Institute Inc.—are today deliv-ering canned support for XML/A, butwhen Microsoft Corp. finally ships its SQL Server 2005 database nextyear, XML/A will be said to have trulyarrived.

XML for Analysis (XML/A) describes a single API for exchanging databetween OLAP clients and servers—on any platform and in any language.Like all putatively open standards,XML/A has been a long time coming:The first draft specification (developedby Hyperion and Microsoft) was pub-lished in April of 2001.

Microsoft, for its part, has supportedXML/A in its .NET framework for sev-eral years now. The software giant alsopublishes a software development kit(SDK) that lets an XML/A client com-municate with Analysis Services—Microsoft’s OLAP engine—running onSQL Server 2000.

But SQL Server 2005 will boast nativesupport for the XML/A standard,which—if history is any indication—could push XML/A over the top. Afterall, when Microsoft built an OLAPengine into its SQL Server 7.0 data-base, the software giant’s OLE DB forAnalysis API became, almost over-night, a de facto standard. “Microsofthas made XML/A the native API forAnalysis Services in SQL 2005. This ispretty significant in that it basicallyforces the adoption of the technologyby third-party vendors, integrations,and customers,” says Chris Harrington,president of Web application andinformation portal development shopActive Interface Inc. “Everybody noton board with Service OrientedArchitectures (SOA) is going to have tocome up to speed.”

Harrington has been involved withXML/A almost from the beginning, andhas for several years offered XML/Aproofs-of-concept on his Web site,www.activeinterface.com/thinolap.html.

The attraction of XML/A, sayHarrington and other users, is that it

provides a more or less seamless way toturn unconventional clients—such asMicrosoft Excel, Microsoft PowerPoint,or even a vanilla Web browser—intoOLAP front-end tools. In this respect,they argue, it’s an ideal glue for the SOAsthat some organizations are building.

XML/A enables a “course-granular,loosely coupled approach to information[that] really makes possible the freeingof that information within the enter-prise,” argues Harrington. “XML/A andother Web service APIs turn informa-tion into a resource which can be identi-fied via a URL. What you do with theinformation obtained from that URL islimited only by your imagination.”

Some companies are already tappingXML/A to support promising uses.Consider Australian advertising agencyY&R and Wunderman, which—with a helping hand from consulting outfit Maximise Consulting Pty Ltd—implemented Microsoft’s XML/Aprovider to expose OLAP capabilitiesto ad teams scattered across theAustralian continent.

“Ad-campaign teams are divided up all around the country. They includemembers such as the analytical team,the client, the campaign managers,[and] other contractors,” explainsMaximise consultant Scott Underhill.

“In the past, they have used databasesystems, which has [resulted] in data

Making the Case for XML/A

I

This article appeared in the December 1, 2004 issue of the e-newletterBI This Week.

by Stephen Swoyer

35VOLUME 2

THE CASE FOR XML/A

being pulled from the system, thenused in tools such as Excel, and thensent around to other members of theteam through Excel and PowerPoint,”he says.

This approach was kludgey, however.Excel, PowerPoint, and other MicrosoftOffice applications simply weren’tdesigned for use in distributed environ-ments, and they certainly weren’t devel-oped with SOAs in mind. So Y&Rturned to Maximise, which tappedXML/A to graft an SOA onto its exist-ing fat client infrastructure.

“What we have done with the XML/Atechnology is to allow [users] to createtheir own views of the data through dif-ferent client applications,” Underhillsays. “The major client application isthe Web with the use of BI-Controls.We allow for all the data on the systemto be viewed through a Web page, [and]the data available will depend on who[is] logged on to the system.”

The result, Underhill continues, isspeedier—and more transparent—access to OLAP services. “This hashelped Y&R greatly, as specific views ofthe data can be set up on a Web page,then they have the freedom to save theview. Next time their client comes andlooks at the data, they can see the viewsthat have been set up. But, not only dowe have it set up on the Web page; Y&Ruses XML/A to talk directly to the cubefrom Excel and other applications.”

In lieu of Microsoft’s XML/A provider,Underhill says he could have used anyof several different SOAP providers, butthis would probably amount to graftinga kludge on top of a kludge, he main-tains: “There are older Web connectorswhich we have used and still use tosome degree. These older SOAP con-nections don’t provide the flexibilityand security that XML/A provided.XML/A seems to talk to MicrosoftOffice products a whole lot better withlittle effort.”

BI architect Mark Morris says hisorganization is using XML/A to recastone of its most important financial andcustomer targeting applications as anSOA-friendly play. Like other OLAPdevelopers, Morris has traditionallytapped a patchwork of Microsoft tech-nologies, such as OLEDB for Analysisand Pivot Table Services, along withADOMD.NET, to accomplish some ofthe same tasks.

So why use XML/A? Partly, Morrissays, because of Microsoft’s obvioussupport for the standard. Just as impor-tant, however, is OLE DB for Analysis’limited scalability: It’s based onMicrosoft’s proprietary COM architec-ture, and is poorly suited for the highlydistributed world of Web services.“XML/A has been finally chosen by usdue to a number of factors. Theseinclude the forthcoming AnalysisServices 2005 support for XML/A andexcellent scalability,” he comments.

Morris’ organization isn’t moving com-pletely away from OLE DB forAnalysis, however. “Although XML/Awas chosen for the application’s pri-mary functionality, we do continue todeploy OLE DB for Analysis [andPivot Table Services] to support ourMicrosoft Office user base,” he con-firms. “These users utilize a mixture ofExcel and Office Web Controls fortheir custom analyses. XML/A is exclu-sively used by our internally developedBI portal and has proven to date to bearchitecturally sound.”

BI architect Harrington, for his part,believes it’s only a matter of time untilXML/A gains truly broad adoption.“My own crystal ball tells me that all ITplatforms are moving in the direction ofWeb services, and so whether it isXML/A or some other XML schema,there will be Web service front-ends—either provided by the vendor or bythird parties,” he comments.

One holdout is Sun, which—not sur-prisingly—backs another Web servicesOLAP standard, Java OLAP (JOLAP).Hyperion itself first joined the JOLAPcamp, and has made noises about sup-porting both standards. Harringtonbelieves that the two worlds can be easily reconciled, however. “Sun’s J2EE is certainly getting with the SOA pro-gram—so why not a SOAP wrapper forJOLAP?” he wonders. “I assume thatthat this is how Hyperion has developedtheir [own] XML/A provider—as awrapper on their JOLAP API.”

In the end, he continues, XML/A—orsomething like it—just offers an easier,more scalable way of exposing OLAPcapabilities. “Remember that the beautyof XML Web services is that we care awhole lot less about how the data is struc-tured,” he says, noting that XML canalways be restructured by means of theeXtensible Stylesheets TransformationLanguage (XSLT). “That is my approachfor transforming XML/A results intoWeb pages, PowerPoints, Excel work-books, etc. A different XML schemadoesn’t change the information flowarchitecture when you use Web servic-es—but the information sources mustexpose Web services to participate in thisnew paradigm.”

Part of XML/A’s attraction, saysMaximise’s Underhill, is that it canenable scenarios beyond seamless com-munication between front-end applica-tions and OLAP engines. “One of ourbiggest problems in data mining is thequality of the data and the client’s abili-ty to normalize the data. Something likeXML/A is definitely a way forwards,”he concludes. “[It] gives us the ability tobrowse the data simply and create ourown normalized views for modeling.” n

Stephen Swoyer is a technology writer based in Athens, GA. He can be reachedat swoyerse@percipientanalytics.com.

Project8 2/16/05 2:36 PM Page 1

CASE STUDY

37VOLUME 2

The job of the Ford Motor Co. Business Assistance

Center (BAC) is to field inquiries from roughly 5,500

Ford dealerships in the U.S. and Canada, as well as

hundreds of car rental agencies and other fleet oper-

ators. More than 300 call center agents are trained to

provide accurate, timely information to dealer ques-

tions that range from warranty coverage to market-

ing campaigns to training certifications.

But when it came time for the BAC, based in Dearborn,MI, to report to Ford managers on call center statistics, per-formance, and hot-button business issues, the informationwas not always timely and not always accurate.

The reason: Ford’s BAC relied on techniques and technologyabout as antiquated as an Edsel. Two consultants workedpractically full time to sort through information thatspanned seven call centers and three operational systems,and manually keyed the data into Excel spreadsheets.Hundreds of voluminous paper and electronic Excel reportswould be distributed to managers—full of data that wasmany weeks old.

With its call center reporting clearly in need of a tune-up, Ford’s BAC found a solution with IlumifiAPF Inc., a spin-off of Brooks Automation Inc. created in late 2001 tobring Brooks’ APF (Advance Productivity Family) reporting

software for the semiconductor industry to the mainstreambusiness market.

First deployed as a pilot in late 2001 and now in full production, APF has enabled Ford to reduce from weeks todays the time required to produce reports on call center statistics, performance, and top business issues. Labor costs have been reduced by more than one full-time equiv-alent employee.

Ford has realized new visibility into its BAC operationswith data that is markedly more reliable and consistent thanin the past. With it, call center managers are better able tomonitor center and agent performance, identify weakness-es, perform root-cause analyses, and implement improve-ments. Separate reports distributed to about three dozenbusiness unit managers and analysts track issues by warran-ty claims, vehicle order processing, training, and other met-rics. A key to these dividends is combining data from threediscrete operational sources that record business aspects of acall, call statistics (such as duration and hold time), andquality of performance.

“It’s quicker, more accurate because you don’t have humanintervention, and we save money because we don’t have twopeople laboring to do this,” says Steve Rinaldi, the BAC sys-tems manager. “It’s paid for itself very quickly. We can getout reports with metrics that show how we’re running thebusiness, and supply the business units with relevant oper-ational data they need in a very quick manner.”

BI Case Study: Ford’s Business Assistance Center

Hits the Fast Lane with IlumifiAPF

This article appeared in the Winter 2004 Business Intelligence Journal.

by Mark Hammond

CASE STUDY

38 TDWI’S BEST OF BUSINESS INTELLIGENCE

Semiconductor Specialist Aims for theMainstream Business MarketBrooks, a $300 million company based in Chelmsford,MA, supplies hardware and software to chip manufacturers.It had long sold APF to semiconductor companies such asIntel, IBM, and Samsung, largely to support real-timereporting on manufacturing operations. With an installedbase of more than 150 APF customers, Brooks bargainedthat its reporting software was ready for prime time, partic-ularly for the call center.

The first installation was at Brooks itself; the companyadapted its semiconductor-oriented APF software forreporting against its Siebel-based call center data. In summer2003, Ford became IlumifiAPF’s first real customer.

Both the APF software and the Ford implementation aresomewhat at odds with the prevailing trend in the businessintelligence industry. Unlike most query and reportingtools, APF is not available in a Web-based version; itrequires a client installation to build and run reports.Officials at IlumifiAPF, based in Pittsburgh, PA, say that aWeb-enabled edition is in the works.

As a result, the Ford BAC implementation does not providebusiness users with ad hoc query and reporting capabilities.Though reports are viewable over the Web, business man-agers who want a special report must route requests throughthe IT group. In a sense, Ford is challenging the conven-tional wisdom that end-user empowerment is a good thing.

“Ford’s philosophy is to keep reporting in the hands of ITto help ensure consistency,” says Ron Weber, Ford’s manag-er of vehicle order processing. “We might be making it eas-ier for end users, but we also want to use the tool to driveconsistency across the centers. If we have centers generatingsix or seven reports on their own, we say let’s do it across theboard because other centers maybe can benefit from them,too.”

Ease of Data Mapping and ETL: Addressing a “Real Pain”Ford’s BAC is an aggregation of seven call centers, each spe-cific to business and functional areas. These centers handleinquiries and issues on vehicle order processing, marketingcommunications, warranty coverage, training, and certifica-tion, e-commerce Web support, corporate support, and satel-lite-based data communications between Ford and dealers.

With more than one million calls a year (plus an increasingvolume of e-mail), Rinaldi says assembling reports was nosmall order. “It was a real pain to go into all those systemsand pull all that data together,” he says.

Supporting the seven call centers are three operational sys-tems. A Siebel Systems call center application tracks thebusiness nature of a call, such as vehicle ordering issues anddealer training. An application from Avaya Inc. records dataon the call, such as how long a call lasts, the caller’s waittime before pick-up, the number of call transfers, and soforth. A third system from etalk Corp. provides servicequality observation.

APF’s built-in data mapping and extraction, transforma-tion, and loading (ETL)technology has proved to be itsforte, Rinaldi says. A graphical, drag-and-drop interfacemakes it simple to identify sources, extract data, and mergeit into the APF repository. APF’s mapping tool reads thehost database schema, creates a metadata file, and mapsdata between source and repository with automatic linkingthat eases difficulties often encountered with table joins.

“The major reason we selected them was because most tra-ditional data warehousing products require you to do a lotof work up front to define what your outputs are going tolook like,” Rinaldi says. “This system is completely differ-ent. You don’t have to know what your outputs will looklike because Ilumifi handles that. You don’t have to gothrough a detailed analytic exercise.”

Unique Column-Based Repository Supports High PerformanceAPF is also unique in its use of a proprietary, column-basedrepository that assumes the role conventionally served by anRDBMS-based data warehouse. Unlike the field and tablestructure of a relational database, the APF repository storesdata in self-indexing column structures versus the rows-and-tables structure of an RDBMS.

Aimed at high-performance analytics, the net effects of col-umn-based data stores are to accelerate query speeds, pro-vide storage of detailed data (versus summary data), andcompress data through tokenization to minimize storageoverhead. Other column-based repositories includeNucleus from Sand Technology Inc., Sybase IQ fromSybase Inc., Kdb+ from KX Systems Inc., as well as prod-ucts from Alterian Inc.

Because the APF repository stores detail data in a compactmanner, the Ford BAC will be able to collect and maintaininformation dating back many years, Rinaldi says. Ford exe-cutes nightly extracts of about 60 MB, and with tokenization,some 4 million service records and 7 million activity recordsamount to just 8 GB of data in the IlumifiAPF repository.

Before APF, Rinaldi notes, the BAC had no means of storinghistorical data generated in the Avaya application; it would

CASE STUDY

39VOLUME 2

“roll off ” every month or so. With the new system, analystswill be able to examine Avaya information collected overmonths and years to better understand long-term trends.

The column-based repository stores data in a “temporal”manner—that is, each variable is recorded and versionedover time. “It keeps even more than a mirror image of thedata from the applications because it stores a history of thedata,” Rinaldi says. “It’s a snapshot of what your data wasone day, the day before that, and what it is the next day.”

User Wish List: Better Front-End Reporting, MetadataDespite its strengths in data mapping, ETL, and data struc-ture, Rinaldi says IlumifiAPF could use improvements in itsfront-end reporting and semantic metadata capabilities. “Itsextraction capabilities are better than its end-reportingcapabilities,” Rinaldi says. “Sometimes we’ve found it a lit-tle tedious trying to figure out how to present something onthe report side. Though it can be a little difficult on thefront end, it does have some very nice presentation toolswith charts and graphs.”

Similarly, metadata is not treated semantically or in termsrelevant to a non-technical user. “End users who are not asknowledgeable about our database structure may be won-dering what a particular field is if they do reporting forthemselves,” Rinaldi notes.

Outweighing those weaknesses is a consideration increas-ingly important to many organizations today—cost.Rinaldi would not disclose Ford’s investment in IlumifiAPF

but did characterize it as a “good value.” The fact that itincludes built-in ETL and data repository—componentsoften purchased separately for a data warehousing environ-ment for hundreds of thousands of dollars—made it all themore attractive.

IlumifiAPF officials say, however, that the product may belicensed for as little as $150,000, including implementationservices and training.

Another of the product’s strengths is ease of implementa-tion; the pilot was up and running in a matter of weeks.Given the low development and deployment overhead,Rinaldi and Weber say Ford may broaden APF deploymentsto more user desktops once an upgrade from Windows 98to Windows 2000 is complete.

They also envision providing greater commonality for BACreporting. “Call centers do continue to generate ad hocreports, and we’re saying why don’t we standardize on acommon system and let Ilumifi pull the data,” Weber says.“It really drives you to look for other opportunities. We’rediscovering more and more applications for it—it’s like thesky’s the limit.” n

Mark Hammond is a freelance technology writer based in San Francisco who specializes in data warehousing and business intelligence. He can be reached at mfhammond@earthlink.net.

Project3 1/25/05 12:38 PM Page 1

41VOLUME 2

ALIGNING BUSINESS AND IT

n spite of all the talk about the need to align business and the information technol-ogy (IT) department, little progress is being made. Part of the problem is systemicto IT departments and technical people, but another part involves the business’ will-

ingness to constructively engage with IT on a long-term basis.

Fear and Loathing...In researching dashboard and scorecard applications, I’ve come across an inordinatenumber of companies where a pitched battle is raging between business and IT. In mostcases, the problem arises because a business group needs to transition its highly success-ful dashboard project to centralized IT, so the rest of the company can reap the benefits.

The business group is terrified about ceding control over the design, architecture, andbudget of its pet project to a central IT group, which it views as slow, incompetent, anduncompromising. The business sponsors cite numerous examples of IT ineptitude toreinforce their notions that the IT department will suck the lifeblood out of the projectand leave it to die a slow, inexorable death.

But the IT group views the business as a spoiled child that wants what it wants when itwants it and won’t wait for IT to lay the necessary foundation to ensure the long-termsuccess of the system. It is also bitter that the business expects it to deliver an ever-increasing number of “high-priority” projects in shorter time frames while reducingcosts, shrinking staff, standardizing suppliers, and living under the constant threat ofoutsourcing and offshoring.

The result is a tense standoff in which each group fulfills the other’s worst perceptions ofitself. If the business has the upper hand, it will maintain control of the technical aspectsof the project, creating another non-integrated system that will be costly to maintain inthe long run. If IT gains control, it will halt development of new end-user functionalityuntil it brings the infrastructure into conformance with its architectural standards.

So what can be done to slice through this Gordian knot? What will it take for both sidesto enter into a relationship of mutual respect? Like a marriage on the rocks, business andIT need some serious counseling before they can work together effectively. Part of thecounseling involves taking a number of baby steps that improve communication andbridge mutual distrust by helping each side better understand the other’s challenges and dilemmas.

It Takes Two to Tango

This article appeared in TDWI’s Case Studies & Solutions e-newsletter,November 18, 2004. Visit www.101com.com/subs_news.asp to subscribe.

by Wayne Eckerson

I

42

ALIGNING BUSINESS AND IT

TDWI’S BEST OF BUSINESS INTELLIGENCE

Counseling for ITDuring the past 10 years, IT has come torecognize its job is not to deliver tech-nology for technology’s sake but to pro-vide exquisite service to its customer—the business. Like an alcoholic who pub-licly admits the problem, this is a step inthe right direction. But it’s only the firststep. Verbal acknowledgement alonedoesn’t translate into remedial action.

Systemic Dilemma. To take the nextstep, IT must translate goodwill intoaction. The following questions can helpdetermine whether your IT team is pay-ing lip service to serving the business tak-ing action. If you can respond positivelyto most of the questions below, you areon the right path.

Does your team:

• Read the same trade magazines as itsbusiness counterparts?

• Attend the same conferences as itsbusiness counterparts?

• Go to lunch regularly with its businesscounterparts?

• Read the company’s annual report?

• Read the short- and long-term strategicplans for your company?

• Understand the complete businessprocess that encompasses the applica-tion they’re developing?

• Have a majority of members with morethan 10 years of experience in yourcompany’s industry?

• Have a majority of members with both acertification as a database administra-tor and a master’s degree in businessadministration?

What better way to align with the busi-ness than to eat, sleep, and breathe like abusiness person?

Unfortunately, the central IT depart-ment—by virtue of its being a separateorganization within the company—often functions as a subculture with

its own rules. Central IT groups havetheir own jargon, incentives, reportingstructure, and career paths, which aredifferent from the business it serves.

In contrast, departmental or line ofbusiness technical staff often enjoy amuch healthier relationship to theirbusiness users than central IT. Why?Rather than existing in a technical sub-culture, these “embedded” IT staff sitside by side with the business people andfunction as a single team with the samegoals and ambitions.

Collaborative Strategy. I recentlyattended a presentation by an IT managerat a major Blue Cross Blue Shield organ-ization who had developed a strategicplan to foster a more collaborative part-nership with the business.

There are five key initiatives in his plan.One of the more innovative is the devel-opment of an application architecturethat evaluates current and proposed busi-ness applications by both business fit (asdefined by business) and architectural fit(as defined by the IT group). Eachgroup’s evaluation for all applications isdepicted in an easy-to-read quadrantchart that plots business fit on the x axisand architectural (technical) fit on the yaxis. (See Illustration 1.) Applications inthe lower left quadrant are candidates forelimination or consolidation; applica-tions in the upper right represent an

optimal fit from both a business andtechnical perspective.

Applications in the remaining two quad-rants—lower right and upper left—needmodification before they meet businessor IT requirements for sustainability.The ongoing exercise of evaluating appli-cations in this manner is a wonderfulway for both groups to communicatetheir requirements and view them in acomplementary and interdependent way.

Counseling for BusinessAlthough IT groups generally get thelion’s share of the blame for misalign-ment between business and IT, it takestwo to tango, as they say. Business sharesequal blame for the frustration that itfeels towards IT—perhaps more, becauseit doesn’t always recognize how itsactions and behavior have contributed tothe problem.

The problem is that business changes toofast for IT to keep up. It harbors a short-term bias toward action and rarely takesa long-term view toward building sus-tainable value. This is especially true inU.S. companies which, true to theirWild West heritage, are notorious foracting first and asking questions later.

Decentralized organizations magnify thisbehavior, parceling out authority to divi-sions and departments to make decisionsfaster and in the context of local markets.While there are advantages to decentral-ization, considerable downsides still con-tribute to the perpetual misalignment ofthe business and IT. The scores of ana-lytic and operational silos, including thehundreds and thousands of perniciousspreadmarts that hamstring corporateproductivity, testify to the business’ fixa-tion with speed and decentralized deci-sion making.

Project Prioritization. Finally, thebusiness has the upper hand in its rela-tionship with IT and it often rules in ahigh-handed and irresponsible manner.

Improve Business Fit

Maintain/Evolve

Phase Out/Replace

Improve Technical Fit

0% 50% 100%Business Fit

Tech

nica

l Fit

0%50

%10

0%

Application Scorecard Framework

Illustration 1. The quadrant chart above is anexcellent tool to help business and IT begin tocommunicate their needs and requirements ina more proactive, positive manner.

In many organizations, executives threat-en to outsource or offshore IT when itdoesn’t deliver sufficient value, ignoringhow their own actions and decisionscripple IT’s ability to function effective-ly. The business often lacks a reasonabledegree of restraint and self-disciplinewhen it comes to IT projects. One ITmanager I talked to recently said theirannual planning process is a shambecause the business can’t discipline itselfto live within its limits.

“At the beginning of every budget year,the business prioritizes IT projects forthe coming year. Out of 90 projects,they score 60 of them as ‘high priority’and we schedule them,” says the belea-guered IT manager. “But even beforeJanuary 1st arrives, the business adds20 more ‘high-priority’ projects to our list and adds another 20 projectsbefore April. And then they tell us inMarch that we are already two monthsbehind schedule!”

The IT manager said he had negotiateda new project prioritization processwith the business that in effect requiredthe business to operate in a “zero-sum”environment. If they added projectsafter the budget, they needed to cutothers. Although the IT manager washopeful the new program would suc-ceed, he also half-joked that he may notbe around in a year or two if he has totell the business repeatedly to abide bythe new guidelines.

Portfolio Planning. Another way tohelp the business live within its means isto establish an analytic applications port-folio that shows how IT can deliver aseries of related applications built on acommon infrastructure over a condensedperiod of time, such as 18 to 24 months.

Jill Dyché, partner at Baseline Consultingin Sherman Oaks, CA, created thechart in Illustration 2 to help business

executives understand the iterativeprocess of building analytic applica-tions and how they can accelerate theprocess if they want to pay the cost ofcreating parallel development teams.

The chart shows executives that they canget everything they want by building ona common infrastructure instead ofadopting the “go it alone” approach. Ifthey want their applications faster, theycan pay for parallel development teams.This practice shields IT from accusationsthat it works too slowly, leaving decisionsabout speed and cost to the business,where they belong.

ConclusionThe ideal way to align business and IT isto combine knowledge of both disci-plines in one person. While this type ofindividual is a rare commodity today, wewill increasingly see people with bothbusiness and technical literacies in thegenerations to come. Until that time,business and IT will be left to figure outstrategies and techniques to collaboratein more fruitful ways.

The key for now is to resist tendencies tocultivate a purely technical culture with-in central IT. The IT group must not bea cocoon for IT professionals who canavoid contact with the business usersthey’re trying to serve without completeknowledge of the strategy and processestheir applications and systems aredesigned to support.

At the same time, business needs tounderstand that Rome wasn’t built in aday. And while they may use their ownmoney and technology to chase immedi-ate opportunities, they need to acknowl-edge that these victories will be short-lived if there is no scalable or sustainableway to transform the initiatives intoenterprise resources built on a solidarchitectural foundation.

We still have a long way to go towardaligning business and IT, and there ismuch work to be done. n

Wayne Eckerson is Director of Research at TDWI. He can be reached at weckerson@tdwi.org.

43VOLUME 2

ALIGNING BUSINESS AND IT

Analytic Application Delivery Portfolio

Illustration 2. One company’s analytic application portfolio delivery schedule. This tool makes iteasier for executives to see their needs will be met over the long term by building on a standardinfrastructure, and lets them decide how much money they want to spend to accelerate development.

Project6 1/31/05 1:08 PM Page 1

SALARY SURVEY

45VOLUME 2

TDWI 2004 Salary, Roles, andResponsibilities Report Summary

ata warehousing salaries appear to be recovering fromthe dot-com implosion and economic stagnation.Average salaries increased by 3.2 percent, from

$82,977 in 2002 to $85,619 in 2003. This reverses a declinebetween 2001 and 2002, when of hundreds of dot-coms dis-appeared—and with them, thousands of six-figure salaries.

Nearly half (46 percent) of data warehousing professionalsreceived salary increases of 4 percent or more. These risingwages probably contributed to a decline in the number ofpeople who work a second job, down from 20 percent in 2002to 13 percent in 2003. The percentage of people receivingstock options (27 percent) remained largely unchanged.

Business intelligence and data warehousing professionals con-tinue to enjoy greater salaries than many of their IT peers.TDWI survey data shows the median salary (both staff andmanagers) at $83,000 for 2003. In comparison, the mediansalary is $65,000 for a general IT staffer and $89,000 for anIT manager, according to the InformationWeek annual salarysurvey published in April 2003.

D2003 Salaries

2003 2002Average salary $85,619 $82,977Percent receiving options 27% 26%Percent moonlighting 13% 20%

2003 Salary Increases

2003 20020–3% 51% 47%4–5% 27% 26%6–10% 13% 19%11–20% 5% 3%21%+ 1% 2%Declined 3% 3%

The TDWI Salary, Roles, and Responsibilities Report ispublished annually in February as a benefit to TDWIMembers. For information about how to receive thecurrent 2005 report and become a TDWI Member, pleasevisit www.tdwi.org.

SALARY SURVEY

46 TDWI’S BEST OF BUSINESS INTELLIGENCE

Salary by Role, Experience, and CertificationsChief architects and business sponsors—those with the great-est responsibility for the success of a data warehousingproject—received the highest salaries in 2003, averaging sixfigures. Program managers, who supervise multiple projectsand act as liaisons with business sponsors, also crack the six-figure threshold.

Several roles enjoyed substantial salary increases. The surveydata shows the average wages for ETL developers, BI develop-ers, database administrators, and data designers are up roughly$9,000–10,000 over 2002. (Some roles are not cited in thetable below because of a statistically low number of responses.)

In five years, the typical data warehousing professional willearn $85,392. After 10 years, a wage exceeding $100,000 is asafe bet. Entry-level salaries, meanwhile, continued a post-dot-com decline, down 4 percent, from $72,122 in 2002 to$69,193 in 2003.

Generally, it pays to have several certifications in specific technologies. Those with four certifications average morethan $90,000 a year, though that sum declines as more cer-tifications are achieved. More than 41 percent hold at leastone certification.

Salary by Data Warehouse Size and ScopeInterestingly, more bytes means more bucks. Personnelrunning smaller warehouses receive below-average wages,while those involved with systems in the 1 terabyte rangeenjoy compensation well above the norm. Similarly, salariestend to scale up as the number of source systems, active users,and subject areas increases.

Average Salary by Role

$105,970 Chief Architect$103,372 Business Sponsor/Driver$101,329 Program Manager$87,348 Project Manager$83,353 Data Designer/Modeler$83,316 Technical Architect$81,754 Decision Support (BI Tools) Developer/Manager$79,987 Data Acquisition (ETL)Developer/Manager$76,158 Business Requirements Analyst$73,707 Database Administrator$72,644 Data Warehouse Administrator

Average Salary by Years of Experience0 Years $69,193

1 Year $74,182

2–3 Years $76,019

4–6 Years $86,007

7–9 Years $97,676

10+ Years $102,802

Average Salary by Certifications0 Certifications $85,809

1 Certification $84,863

2 Certifications $85,982

3 Certifications $89,919

4 Certifications $90,301

5+ Certifications $83,856

Average Salary by Data Warehouse Size<1GB $79,105

1GB–100GB $80,690

100GB–500GB $83,021

500GB–1TB $92,049

1TB–10TB $91,276

10TB+ $91,566

Average Salary by Number of Data Warehouse Source Systems

1 Source $76,151

2–3 Sources $81,009

4–6 Sources $84,020

7–10 Sources $90,177

11–20 Sources $91,026

21–50 Sources $91,439

50+ Sources $95,894

Average Salary by Number of Active Data Warehouse Users

1–20 Users $81,458

20–100 Users $83,845

100–250 Users $86,576

250–500 Users $87,818

500–1,000 Users $84,390

1,000–2,500 Users $87,575

2,500+ Users $90,303

Average Salary by Number of Subject Areas the Data Warehouse Supports

1 Area $82,164

2–3 Areas $83,302

4–6 Areas $84,889

7–10 Areas $90,042

10+ Areas $87,242

SALARY SURVEY

47VOLUME 2

Salary by Data Warehouse Team/Budget andCompany SizePeople working on larger data warehouse teams and withlarger budgets tend to earn higher salaries. The differential isespecially striking for average salaries by overall team budget,with a gap of nearly $31,000 between low and high averagesalaries.

But a bigger company does not necessarily mean a biggersalary. The highest salaries are found at the smallest companies(fewer than 100 employees) and the largest companies. Smallto mid-size firms offer the lowest wages—an average of$79,362 for those with between 500 and 999 employees.

Salary by Years of Service and Purchasing AuthorityLongevity with a single company is not necessarilyrewarded in salary. The data reveals that people with onlythree to five years of service to one company earn virtuallythe same as (or even more than) those with a great dealmore tenure. Similarly, a person with less than a year in hisor her position is likely to earn the same as a person withfour years in his or her position.

Not surprisingly, managers with purchasing authority (8percent of respondents) are better compensated than thosewho specify/recommend products (69 percent) or determineneed (23 percent).

Salary by U.S. Region and IndustrySalaries vary widely by geographic location and type of indus-try. A data warehousing professional working in the South in

Salary by Years at Current Company0–1 Year $83,749

2 Years $84,011

3–5 Years $86,125

6–10 Years $87,091

11–20 Years $86,830

20+ Years $84,567

Average Salary by Years in Current Position0–1 Year $83,814

2 Years $82,824

3 Years $87,810

4 Years $83,584

5+ Years $89,095

Average Salary by Purchasing AuthorityDetermine need $77,292

Specify/recommend products or services $86,468

Final purchasing authority $102,758

Average Salary by Data Warehouse Team Budget<$100,000 $76,671

$100,0000–$500,000 $84,093

$500,000–$1 million $91,966

$1 million–$5 million $99,377

$5 million+ $107,506

Average Salary by Data Warehouse Team Size1–3 FTEs (full-time employees) $78,368

4–6 FTEs $83,888

7–10 FTEs $92,760

11–20 FTEs $90,704

20+ FTEs $95,513

Average Salary by Company Size<100 Employees $89,865

100–499 Employees $83,576

500–999 Employees $79,362

1,000–9,999 Employees $85,902

10,000+ Employees $90,214

Average Salary by U.S. RegionNew England $92,533

Pacific $92,428

Mid-Atlantic $90,503

Southwest $84,521

Midwest $80,939

Central Plains $79,963

South $79,573

Average Salary by Top IndustriesPharmaceuticals $103,594

Consulting/Professional Services $96,437

Software/Internet $96,284

Financial Services $89,807

Manufacturing (Non-Computer) $85,544

Retail/Wholesale/Distribution $83,812

Telecommunications $82,667

Insurance $81,120

Healthcare $77,589

Government (Federal) $77,164

Education $72,454

Government (State/Local) $67,620

the education field averages $69,500 a year; a WestCoast–based consultant averages $115,000 a year. Data ware-housing professionals employed by pharmaceutical manufac-turers appear especially well compensated.

Salaries are on the rise in the Pacific region, up 11 percent overlast year to $92,428. The chart “Average Salary by TopIndustries” ranks salaries for the 12 industries with the mostsurvey respondents.

Salary by Age and GenderData warehousing is a profession dominated by young men.Seventy-one percent of respondents were age 45 or younger;only 7 percent of respondents were age 56 or older. Salariestend to peak in one’s later years, with an average of $91,407for those age 56 and older.

The number of female data warehousing professionals isrising, however, from 27 percent in 2002 to 32 percent in2003. And salaries for women are up as well, increasing 5.2percent over 2002 to an average of $79,558. Men continue toearn more at $88,488, though the margin between male andfemale salaries shrank from 14 percent to 11 percent between2002 and 2003.

Women increasingly are holding positions of greater authorityand compensation. The number of women serving as chief

architect increased from 10 percent to 15 percent between2002 and 2003, though they earn considerably less than men($93,500 versus $108,180). The number of female businesssponsors grew from 9 percent to 20 percent, though they tooearn less than men ($95,500 versus $105,292). Thirty percentof program managers are women—very nearly equalingwomen’s representation in the data warehousing field at large.

ExperienceMany data warehousing professionals are relative newcomersto the field and their employers. Some 38 percent of respon-dents have three years or less of experience, and 43 percenthave been with their employer for three years or less. Morethan two-thirds have spent no more than three years in theircurrent position. n

Average Salary by Age

<25 $58,09426–35 $77,68636–45 $90,15746–55 $89,09256–65 $91,407

Industry Profile by Age

Average Salary by Gender

Men $88,488Women $79,558

Industry Profile by Gender

Men 68%Women 32%

26–35 30%

36–45 39%

46–55 21%

56–65 6%

66+ <1%21–25 2%

Years of Data Warehousing Experience

Years at Current Company

Years in Current Position

2–3 25%

4–6 34%

7–9 15%

10+ 13% 0–1 13%

2–3 28%

4–6 24%

7–10 13%

11–20 13%

0–1 15%

2–3 43%

4–6 21%

7+ 11%0–1 25%

20+ 8%

TDWI’S BEST OF BUSINESS INTELLIGENCE48

SALARY SURVEY

ABOUT TDWI

49VOLUME 2

Mission The Data Warehousing Institute™ (TDWI), a division of101communications, is the premier provider of in-depth,high-quality education and training in the business intelli-gence (BI)and data warehousing (DW) industry. TDWI isdedicated to educating business and information technologyprofessionals about the strategies, techniques, and tools thathelp professionals successfully design, execute, and maintainBI and DW projects. TDWI also fosters research and con-tributes to knowledge transfer and the professional develop-ment of its Members. TDWI sponsors and promotes aworldwide membership program; quarterly educational con-ferences; regional educational seminars; onsite courses; solu-tion provider partnerships; awards programs for best practicesand leadership; rich and varied publications; an in-depthresearch program; and a comprehensive, multi-functionalWeb site.

Membership As the BI and DW field evolves and develops, it is necessaryfor professionals to connect and interact with one another.TDWI provides these professionals with the opportunity tolearn from each other, network, share ideas, and respond as acollective whole to the challenges and opportunities in theindustry.

Membership with TDWI allows these professionals to makepositive contributions to the industry and advance profession-ally at the same time. TDWI Members benefit from increasedknowledge of the trends in BI and DW, which makes TDWIMembers some of the most valuable professionals in theindustry. TDWI Members are able to avoid common pitfalls,learn fundamentals quickly, and network with peers andindustry experts to give their projects and companies a com-petitive edge in deploying BI and DW solutions.

TDWI’s Membership includes more than 5,000 people whoare BI, DW, and information technology (IT) professionalsfrom Fortune 1,000 corporations, consulting organizations,and governments in 45 countries.

Membership with TDWI is available to all BI, DW, and ITprofessionals for an annual fee of $249 ($299 outside theU.S.). TDWI also offers Team Membership for organizationsthat register five or more individuals as TDWI Members.

General Membership Inquiries: TDWI Membership5200 Southcenter Blvd., Suite 250Seattle, WA 98188 Tel: 206.246.5059, ext. 113Fax: 206.246.5952E-mail: membership@tdwi.orgWeb: www.tdwi.org

About TDWI

Summary of TDWI Member Benefits:

• Quarterly Business Intelligence Journal

• Password-protected Members-only content to accessTDWI research, publications, and archives

• Annual Salary, Roles, and Responsibilities Report

• Quarterly Ten Mistakes to Avoid series

• Biweekly TDWI FlashPoint e-newsletter

• Printed TDWI in-depth research reports

• Quarterly Member Newsletter

• Special discounts on all TDWI conferences and seminars

• A 15-percent discount on all industry books and coursebooks at TDWI conferences

• A 15-percent discount on TDWI merchandise

TDWI Value-Added Resources:

• TDWI Online: featuring comprehensive event information, e-newsletters, downloadable research, Webinars, whitepapers, and Web-based communities

• Semiannual What Works: Best Practices in BusinessIntelligence and Data Warehousing corporate case study compendium

• Market Solutions comprehensive product and service guide

• Annual technology poster

ABOUT TDWI

50 TDWI’S BEST OF BUSINESS INTELLIGENCE

Team Membership Inquiries:Margaret Ikeda, Member Services SpecialistTel: 206.246.5059, ext. 113E-mail: mikeda@tdwi.org

Education and Training Conferences, Seminar Series, and Onsite Courses TDWI offers educational programs throughout NorthAmerica that inform IT professionals about industry bestpractices and the effective use of business information. Theseeducational programs include quarterly U.S. conferences,regional educational seminars, and onsite courses.

TDWI has developed a comprehensive BI/DW curriculum,and its faculty includes the highest-rated instructors in theindustry—most of whom are well-known consultants, practi-tioners, and recognized experts. TDWI has helped to educatemore than 15,000 IT professionals and business users aboutBI and DW throughout the past four years.

During these events, IT professionals share real-world experi-ences, lessons learned, and tips and techniques they have usedto exploit corporate information for competitive advantage.No other organization provides such educational opportuni-ties. TDWI educational programs provide the education andknowledge for Members to quickly understand and begin suc-cessful data warehousing initiatives.

Certificate Courses TDWI’s Certificate Courses provide attendees with a recordof completion of a TDWI designed and developed course. ATDWI Certificate provides proof that an attendee has com-pleted the course and is prepared for further education or tobegin or join a BI/DW effort for further experience. TDWICertificate Courses can ensure that each member of a teamhas received the same level of education, to avoid confusionon the fundamentals.

TDWI On-TrackTDWI On-Track: A Curriculum Program for BusinessIntelligence and Data Warehousing Professionals enables pro-fessionals to enhance career development through a well-rounded, formalized educational curriculum. Participants inthis program earn credits in clearly defined disciplines ofrequired and elective courses that are offered at TDWI educa-tional events.

CertificationThe Certified Business Intelligence Professional (CBIP)cre-dential, developed and delivered in partnership with theInstitute for Certification of Computing Professionals(ICCP) is a true test-based certification program. The CBIPcredential provides the most meaningful and credible certifi-cation available in the industry. Certification is offered in thesame five areas of specialization as On-Track: Leadership &Management, Business Analytics, Data Analysis & Design,Data Integration, and Administration & Technology. TDWIoffers an ICCP Exam Cram course, which is designed to assistattendees who are preparing to take the examinations, andoffers the exams onsite, at conferences.

TDWI Best Practices Awards The annual TDWI Best Practices Awards identify successfulBI and DW models to follow. Top professionals and analystorganizations assist TDWI in identifying practitioners (indi-viduals who have implemented BI and DW within their com-panies) who exemplify the best approaches to solving variousproblems facing managers and staff.

Leadership Award TDWI recognizes one organization each year that has demon-strated true leadership in BI and DW. The TDWI LeadershipAward winner is selected from the TDWI Best Practiceswinners, who have been evaluated against TDWI Leadershipcriteria. One of these winners receives the Leadership Awardfor the year. This award has quickly become the most covetedin the industry.

Publications and Research TDWI supports, develops, and distributes a wide range ofpublications to keep its Members up to date on the new tech-niques, events, and issues in BI and DW, as well as the trendswithin the vendor community. These publications are avail-able to Members at a special discounted rate and include text-books, journals, reports, white papers, and newsletters inpaper and electronic formats. These publications include:

• Business Intelligence Journal, published quarterly, containsarticles on a wide range of topics written by leading visionar-ies in the industry and academia who work to further thepractices of BI and DW.

• TDWI FlashPoint e-newsletter, published biweekly, featurespertinent, vendor-neutral BI/DW articles written by industryexperts.

• TDWI Member Newsletter, published quarterly, examinescurrent topics from TDWI’s research, education, and vendorpartner programs, including events.

ABOUT TDWI

51VOLUME 2

• TDWI Salary, Roles, and Responsibilities Report, publishedannually, is a survey that examines industry professionals’salaries, roles, and responsibilities.

• Ten Mistakes to Avoid series, published quarterly, examinesthe most common mistakes professionals make in developing,implementing, and maintaining BI and DW initiatives.

• Business intelligence and DW books, published periodicallyby other publishers and resold by TDWI, many of which arewritten by TDWI Fellows and Instructors, cover a wide rangeof issues.

• What Works: Best Practices in Business Intelligence andData Warehousing, corporate case study compendium, pub-lished twice each year, is a collection of case studies andlessons from the experts sponsored by TDWI and leadingsolution providers in the BI/DW marketplace.

• TDWI Market Solutions, comprehensive product and service guide.

• Technology poster, published annually, depicts a BI or DWlifecycle and the technologies and products required tosupport it.

Solution Provider Exhibitions TDWI’s quarterly conferences provide a forum in which BIand DW vendors and service providers can demonstrate theirproducts and offerings. To enhance the educational value ofthe exhibits, each exhibitor has the same size booth. This andother TDWI practices have drawn praise from attendees andvendors for showing TDWI’s commitment to provide a levelplaying field and to drive the focus back to content andquality. TDWI exhibitions are reserved solely for conferenceattendees and TDWI Members. This provides more qualitytime for attendees and vendors to meet one-on-one andeducate one another about requirements and offerings.

Solution Provider Programs TDWI works closely with the vendor community to bring itsMembers the most up-to-date and important informationregarding BI and DW solutions. TDWI’s goal is to giveMembers the opportunity for knowledge sharing withvendors, and also the flexibility to decide the level of interac-tion for this exchange. TDWI has worked to create a hype-freeatmosphere at our conferences, and also in our publications,which is conducive to Members’ education and needs, unlikewhat may be seen at other conference exhibitions.

Web Site TDWI Members may take advantage of a growing collectionof reports, white papers, case studies, articles, conference pro-grams, vendor directories, schedules of events, newsletter sub-

scriptions, registration, and much more on the TDWI Website at www.tdwi.org.

Contact TDWI at: 5200 Southcenter Boulevard, Suite 250Seattle, WA 98188-7911Tel: 206.246.5059Fax: 206.246.5952E-mail: info@tdwi.orgWeb: www.tdwi.org

The Business Intelligence and Data Warehousing Industry The BI/DW industry encompasses a host of disciplines andtechnologies used to analyze corporate information, includingdata modeling, data migration and transformation, dataquality, decision support, metadata management, data marts,online analytical processing, database management, datamining, knowledge discovery, closed loop decision support,and various analytic applications, such as customer relation-ship management, sales force automation, balanced score-cards, marketing automation, supply chain management, andvertical industry applications.

101communications Founded in 1998, 101communications is an integrated mediacompany aimed at specialized targets within the informationtechnology community. Our properties range across diverseformats—trade magazines and journals, e-newsletters, confer-ences and seminars, training courseware and associated Web-based services. Target audiences include software developmentteams, networking and communications experts, and seniorinformation executives in major industries, the public sector,and higher education. We pride ourselves on giving our audi-ence the specific solutions they need to manage the informa-tion technology demands of their organizations.

We are organized into seven distinct areas of expertise:Software Development, Business Technologies, WindowsNetworking & Certification, Education Technology,Government Technology, Office Imaging Technologies, andInternational IT.

Reaching more than 1.5 million IT professionals around theworld, our current portfolio includes 9 magazines, 40 confer-ences and events, and extensive digital offerings. For moreinformation on 101communications, visit www.101com.com.