towards an agile framework for business intelligence...

Towards an Agile Framework for Business

Intelligence Projects

M. Prouza*, S. Brodinová, A.M. Tjoa*

*TU Wien, Vienna, Austria

[email protected], [email protected], [email protected]

Abstract - Business Intelligence (BI) solutions provide

decision makers with information about their business and

beyond as a utility to help them to make better decisions for

their business purpose. Establishing such a BI system in a

company can often be very complex and may end up in a

financial disaster due to cancelling the project because of no

valuable output after a specific time. Complex technologies,

different stakeholders from various domains in a company,

and vague requirements are often the initial situation for a

BI project and reasons for delayed project delivery. Thus,

being flexible and adaptable in the project are key factors

for success. An agile approach can be the answer for a

successful BI project. This paper deals with a conceptional

agile framework for such a project. The focus is put on

identifying yet unspecific or unknown aspects during a

starting phase of a BI project.

Keywords – Busines Intelligence; Agile; Agile setup;

Agile Business-Intelligence; Scaled agile;

I. INTRODUCTION

Information is a valuable resource, especially for decision makers within an organization (e.g. a company, government institution) [1]. Business Intelligence (BI) solution targets to generate information from data and provide the information in a proper way to decision makers (i.e. end users) as a utility for their daily business and strategic decisions. A proper definition of BI for this article can be formulated as “…an umbrella term that includes the applications, infrastructure and tools, and best practices that enable access to and analysis of information to improve and optimize decisions and performance.”[2].

Incorporating a proper BI system within an organization is nowadays highly needed to get the required information for decision support. The value of specific information increases when it is related to additional information beyond the own domain. For example, marketing promotions with finance and product sales. Moreover, the time aspect must also be considered in some cases in order to have the right information at the right time is necessary to increase the quality of decisions [3].

Establishing such a BI system in an organization can be challenging due to several reasons [4]. Firstly, stakeholders from different departments have individual interests. Secondly, heterogeneous source systems need to be consolidated and handled in various ways. Setting data from different domains and source systems in relation is

usually not trivial. Moreover, the development team must deal with unclear and vague requirements especially at the beginning of the BI project. Finally, various skillset of the project members is needed to build up a BI system. For handling such a complexity within a project, a conventional plan-driven waterfall method may not be the best approach. An agile approach seems to be a more appropriate way to establish a BI system [3, 5]. Therefore, agile methodologies are often used in a complex and dynamic environment to facilitate on one hand the handling of uncertainties in defined processes, on the other hand to get fast feedback from end users. Fast feedback helps to facilitate that end users get what they really need, which is often hidden behind the requirements [6].

In recent years, agile approaches have been distributed very fast and a lot of frameworks such as Scrum [7], Large-scaled Scrum (LeSS) [8], disciplined agile delivery (DaD) [9], Kanban or Scaled Agile Framework (sAFE) [10] were introduced providing a set of processes, tools, and artifacts for an agile approach for a small project or the whole organization. Nevertheless, BI development cannot be considered as a conventional software project [11] (e.g. transactional system or online shopping platform) due to a complex architecture, the number of integrated tools, involved departments, and people.

The rest of the paper is organized as follows. Section II briefly reviews state-of-the-art of the BI architecture and highlights several important key facts that should be followed in order to achieve an agile project. Section III deals with agile approaches in a BI context. Section IV provides a conceptional framework with the focus on the initial implementation phase of a BI project with an agile approach. Finally, Section V concludes the paper.

II. THE NATURE OF BUSINESS INTELLIGENCE

A valid description of a BI architecture is provided by K. Collier [12] and consists of four broad layers: staging, integration, presentation and visualization. Each layer (i.e. stage) has its own purpose, targets and specific characteristics. The borders between these layers may shift a bit, depending on the concrete developing approach and tools. However, we strongly recommend distinguishing between these layers as it is essential for the agile framework (see Section V).

1544 MIPRO 2020/miproBIS

While the staging layer should be implemented very simply to get data fast, the second layer requires building advanced automated processes and being robust against changes at the same time. Indeed, the integration layer represents the core of the BI architecture where data with business value is kept over time. Traditionally, this layer is usually modelled in third norm technique. However, using the third norm technique does not follow the agile principles as the changes and adaptions of such a model during the project lifetime are very complex and expensive [5]. Therefore, an adaptable modelling technique is recommended for an agile approach. D. Linstedt and M. Olschimke [13] published the DataVault 2.0 modelling approach in 2013. Importantly, they introduced a new modelling approach of the integration layer inspired by the network of neurons. The approach could be described as a combination of the third norm and star schema modelling [14]. The flexibility is also an important characteristic for the information layer as the rules for transforming data into information can frequently change during the project. Finally, heterogeneous analysis and visualization tools, depending on the specific business purpose are typical for the last layer.

DataVault 2.0 generally encourages scalability of a data warehouse and the ability to change, adapt or extend the model during a project with no or reasonable costs. The approach is used for the presented framework where each layer is described in some detail.

A. Staging Layer – Layer 1

The first layer consists of the operational data sources and the staging area where source data are extracted to the data warehouse environment. Data from source systems arrives and technical metadata such as loading date, or the source system are added. If real time analysis is necessary for specific data (e.g. from Internet-of-Things sensors), this layer may also be omitted. The data in this layer are usually archived or deleted after they are loaded into the next layer.

B. Integration Layer – Layer 2

The integration layer (also called raw vault) is the basis of a BI system. The data are loaded from the previous layer using automated Extract-Transform-Loading (ETL) processes and persisted in prepared data structures [13]. The data from different business domains and source systems are integrated into this layer which means they are set in relation and combined. A business-oriented approach is used for integration with business keys describing each business object (e.g. a customer) [13]. The data structures in this layer are based on three basis entity types: hubs, links and satellites. Hubs usually contain a set of business keys an organization has like customer number or product number. Links represent the relations between business keys which are usually transactional data like sales. And entities from type satellite stores contextual data like product name, sales price or customer zip code. Figure 1. represents a simple model with all three entities and their relations. Data in this layer are kept in a raw manner, meaning that only hard rules are applied (e.g. standardize date formats). It is not recommended applying any business rules which

results in changing the meaning of the data [13] which is part of the next layer. In some cases, it may be possible to perform analysis on the raw data but usually end users do not have direct access to the data [13]. Auditability of the reporting system is given due to the concept of technical historization.

C. Information Layer – Layer 3

The goal of the information layer is to conduct data preparations and transformations in order to achieve a comfortable platform of the information for the decision makers and their business purposes. The term information is used by [13] because data are now provided in an interpretable structure which delivers information to end user. Turning data into information is performed by applying business rules on the integration layer. In this layer, a lot of refactoring work comes through a development team and, therefore, this layer needs to be flexibly concepted. It should be emphasized that business rules or other incorporated logics may change over time. Indeed, these rules may even change during implementation due to better understanding of the business, the data and more concrete specification of the requirements.

In order to react on changes fast enough, D. Linstedt and M. Olschimke [13] recommended starting with non-physical views on the integration layer, which is usually fast and cheap to develop, instead of building complex expensive ETL processes. If performance issues require improvement steps on this layer like adding database indices, partitioning or the views are the necessary steps to be performed.

Conventionally information for business analyses is provided in a data mart or information mart, designed with a dimensional modelling technique [14], where specific business performance indicators such as sales and revenue can be analyzed in greater detail. The level of the granularity is defined by dimensions (e.g. states with corresponding regions or departments with sub-units).

In some cases, it might be appropriate to use raw data marts, directly applied on the staging layer. This kind of data marts is used for fast rollout of simple but usable reports which can be used for review, adaptions and further requirements [13]. Raw data marts might be not of a high acceptance for the end users such as decision makers as the quality of the data can be considerably low due to a lack of data cleaning or applying business rules or standards. This is very crucial, for example, in the financial domain due to demand for accurate financial indicators. In contrast to decision makers, data scientists see the advantages of raw marts as they often need access

Figure 1. Hub, link, and satellite entities set in relation

MIPRO 2020/miproBIS 1545

to the data as soon as possible, even if data are raw and unclean.

D. Visualisation Layer – Layer 4

The last layer provides end-users with the prepared information from the previous layers (i.e. data mart or raw mart). The provided information is used for specific analysis (e.g. data mining) and reporting tools like dashboards or OLAP cubes (see more [15]).

III. TOWARD AGILE APPROACH FOR BI PROJECT

The essential idea of agile is to be flexible and fast enough to react on changes during the BI project. Therefore, agile follows an iterative approach aiming at providing potentially shippable functionalities after each iteration, even if such functionalities are not released to production at the moment. In another word, functionality is added continuously after each iteration till project end. A single iteration follows a time-boxed approach and usually lasts from one up to four weeks [7].

The concrete duration may depend on the specific organization and culture, used tools, involved people or the number of teams, and knowledge or project phase. One week is commonly used for the initial development phases to react fast on the frequent changes which are common at the beginning of the BI projects. In contrast, two or three weeks should be considered in later phases. Two weeks provides faster deployment cycles which conclude in faster access to new data for end users [13] and also have benefits when working with multiple individual teams. A three-week iteration approach can have benefits of focusing on design and development in the first week, followed by managing and testing in the second week, and finishing the iteration with the deployment and the acceptance of the delivery in the third week [13]. A four-week iteration approach should be used, when working in multiple individual teams. In such situation, each team has the opportunity to define their individual duration (i.e. one, two, or four weeks), while keeping an overall team iteration length of four weeks.

Figure 2. illustrates a simplified project approach using agile components starting with an overall initial project workshop at the beginning of implementation. Each iteration starts with a planning workshop, where the Product Owner - an agile role who is responsible for the requirements - shows the development team the desired working items (e.g. user stories or developer stories) for the next iteration. These items are already refined, estimated and prioritized in prior workshops with the end users and development team. After all necessary details

are clarified, the development team commits the work for the next iteration and starts with the implementation of the work items followed by short daily stand-up meetings every day between the team members. At the end of an iteration new functionality is presented to the end user by the developers in a review meeting and feedback is taken for further adaptions. The retrospective meeting gives the team the opportunity to inspect themselves, adapt, evolve increasing efficiency, quality, and motivation, during the project lifetime, The outcomes of the review meeting and the retrospective is used as input for the planning meeting for the following iteration [7, 8, 10].

Such iterative approach strongly requires a closer collaboration between all members of a BI project. These principles assure that accurate information is delivered to the right person at the right time, being one of the main goals of a BI system targets. Unsurprisingly, an agile approach seems to be far more suitable building up a BI platform than the traditional waterfall approach [5, 12].

A. Agile Mindset for BI Solution

The main characteristics of the agile mindset are flexibility, collaboration, periodical demonstrations and feedback. For example, K. Collier [16] stated that a very important factor is to involve management as well as end users in the collaboration. Showing and providing useable functionality in regular cycles has a very positive impact on a BI project and increases the trust in the BI project itself and the team members. Main advantages of an agile approach are, periodical demonstration of useable and practical software to end users, and the fast feedback to the development team if the solution follows the expectations and ideas of the end users. If expectations are not met, even in some fine details, an agile approach considers this in form of changes, which can be prioritized and implemented in one of the following iterations.

In traditional waterfall approaches change requests are often not welcome and may imply in a lack of requirements [17]. This is completely contrary compared to an agile mindset [18]. Detail description of the agile approach (e.g. events, team-roles, artefacts) are not part of this paper. Scrum guide [7] provides and good insight into the agile workflow.

B. Agile Requirements

Agile requirements are generally short and precise descriptions of system behavior and user needs, written in the form of so-called user stories [19, 20]. The user stories are the essentials in the agile methodology [20]. Instead of writing long and time-consuming documents, the user stories allow to react fast on the changes of the business rules and other requirements over time. Besides the user stories, regular agile refinement meetings are necessary to detail the requirements, which helps the development team with understanding the business needs in order to design and create a proper solution.

In the context of BI projects, the requirements for all the layers (i.e. staging, integration, and presentations and visualization) are commonly combined in one user story. Usually, the main focus of such user story is put on the description of the visualization layer (see Section II), more

Figure 2. Simplified presentation of an agile approach


specifically on the front end functionalities. Unfortunately, the excessive focus on the visualization layer only can considerably lower the quality of the final solution. A proper method is provided by [5] with introducing developer stories as a special type of requirements for back end (i.e. other three layers) behavior.

C. Developer Stories

Developer stories are not much different from user stories. Their focus is rather on the implementation of at least the staging and integration layer due to the specific characteristics, the used technology, and the needed skills for each layer. Therefore, the implementation of each layer can be a very powerful concept to increase the quality of the final solution [5]. However, the importance of involving the developer stories must be clearly communicated with members of the BI project, especially management. Some member might see this kind of stories and narratives as unnecessary additional work and not being aware of the advantages of such an approach. There are discussions about a direct business value of developer stories for the end users and, therefore, some experts might not consider as agile.

IV. TOWARDS AN AGILE FRAMEWORK

An efficient agile framework, especially at the beginning of a BI project, can be considered as a key success factor. An ill-designed project framework may lead to a lot of coding with no valuable output or mediocre quality. This section presents a framework, which forces fast integration and delivery of components with the flexibility to deal with uncertainties or changes during implementation. For simplicity, we use the notation layer 1,2,3,4 for staging, integration, information or visualization layer.

A. Prepare Development Phase

We strongly recommend including a preparation phase as an initial phase, also known as iteration zero where technical preparations and nontechnical prework is conducted. One or more workshops with the team members and business users (e.g. end users) should be held to get a backlog of user stories which must be prioritized by their business value. A unified vision of the expected outcome and scope must be clear to all team members and stakeholders as it is essential for a successful agile approach. Besides, the technical aspect must also be prepared like creating the infrastructures and configuration of the client software. The end of this phase starts with the first iteration planning meeting. In practice, the quality of requirements is of low quality at the beginning and the necessary information is often incomplete for the development. All these aspects should be communicated to team members as well as stakeholders.

B. Initial Development Phase

The initial development phase of a BI project is challenging – a new development team has to deal with a

bunch of agile requirements where the quality will increase after certain iterations. The team usually has no specific knowledge about the domains and the data they will work with. Moreover, stakeholders tend to ask for results as soon as possible which raises the pressure on the team. Therefore, a proper setting of the agile approach can help the team work efficiently and deliver potentially shippable BI functionality early.

Developing BI functionality requires the implementation on all four layers where each layer depends on the previous one, we highly recommend using the concept of developer stories and splitting the requirements in manageable sizes. This concept helps the developers focusing on the specific characteristics of each layer.

Figure 3. visualizes an illustrative development cycle where new BI functionality is added within two iterations, and each layer must be implemented. Interestingly, Figure 3. shows that Layer 1 with Layer 2 and Layer 3 with Layer 4 must be implemented and tested together within one iteration which is optional but there is a simple and useful logic behind it. Layer 1 and Layer 2 have the same characteristics, following simple patterns which can be automated. In contrast, Layer 3 and Layer 4 are aiming at turning data into information by adding business rules and providing it to the end users with proper analysis tools. The described approach also works well under specific circumstances. Assuming, data are already integrated at the very beginning of a project, Layer 1 and Layer 2 can easily be omitted. If real-time integration for real-time analysis is needed, Layer 1 can be omitted.

The main idea of this approach is to deliver fully operational and independent components as soon as possible and deploy it on a test system where the processes and integration can run in a regular manner. This also is important to gain experience and identify inconsistency early before the next iteration.

C. Dealing with Uncertainty

Uncertainties are usually not welcome in a software project and commonly in any kind of project. Dealing with uncertainty means dealing with the unknown, leading to failures and errors during implementation or testing. Consequently, the satisfaction of not only stakeholder but also team members may decrease during a project. Typical examples of uncertainty in the BI project are a lack of requirements, insufficient data profiling, unknown changing behavior of the data, and hidden business rules.

Figure 3. Development cycles for new BI functionality in an agile

project


Figure 4 demonstrates a suggested solution on how to deal with uncertainties in the BI project. A test system is used to test the functionality implemented in a previous iteration. ETL processes may show unexpected behavior, or the quality of a report is not as good as expected and adaptions are desired for the end user. These issues can be considered in one of the following iterations. Such a solution will work under several assumptions. Firstly, requirements must be split into achievable user and developer stories in order to recognize unpredictable behavior early. Secondly, the length of the development cycle needs to be setup appropriately. Finally, high priority bugs should be fixed as soon as possible, i.e. within the current iteration.

D. End User Collaboration

End user collaboration is an essential part of the agile principles [18] and a significant success factor establishing a BI system. One of the main targets of a BI system is to help end users, such as decision makers to make better decisions or data analyst to improve the quality of data they analyze. Generally, the BI system is used as a utility to make business more efficient due to faster access to actual information about a specific domain. Therefore, not only fast user access but also an active involvement of end users to the development cycle is strongly recommended.

Figure 5 demonstrates the concept of such a collaboration. At the end of each iteration, a review meeting with all stakeholder intents to present the finished work of an iteration. Giving end users the possibility to have access to the components as early as possible. For example, decision makers should have access to the visualization layer (i.e. Layer 4) while data analysts need to check the data in the integration or presentation layer (i.e. Layer 2 or Layer 3) depending on the character of the analysis. This is a proper way to get fast feedback and accordingly correct requirements or change requests due to newly provided information. Such a concept has a great advantage over waiting too long till all layers are fully finished.

E. Scaling Teams

Agile teams are usually small teams between three up to nine members [7]. Some protagonists work with teams up to 15 members [21]. The optimal size depends on the individuals and the environment in a project. We recommend splitting the team into smaller teams of a proper size due to higher communication effort and complexity of coordination. The splitting strategy can vary from a project to a project and should evolve over

time. For example, a layer-based splitting strategy can have two teams, one team working on Layer 1 and Layer 2 and the second team developing Layer 3 and Layer 4. Another common approach is a domain-based strategy where each smaller team works on all layers but with one domain only, e.g. financial domain.

V. CONCLUSION

In this paper, we address common challenges when establishing a BI project. The paper reviews technical and practical aspects of incorporating agile framework in order to manage the difficulties which arise due to a complex BI architecture. The discussed agile fundaments are, for example, predefined processes, events, a clear architecture with four separated, and flexible but robust data structure. We also focus on how to deal with the difficulties which arise in the initial phase of a BI project and which are often carried along the whole project. The recommended solution is, for example, the end user collaboration during project time. This helps to create a single perspective on the expected results for the team members and gives the possibility for direct feedback to the developers about the solution. Several aspects, such as the size of a team or the length of the development cycles, are discussed in order to choose an appropriate setup, depending on the specific characteristics of the BI project.

VI. REFERENCES

[1] E. Turban, Decision support and business

intelligence systems, 9. ed., internat. ed. ed.

Boston, Mass. [u.a.]: Boston, Mass. [u.a.] :

Pearson, 2011.

[2] G. Inc., "BI Glossary," Glossary. [Online].

Available:

https://www.gartner.com/en/information-

technology/glossary/business-intelligence-bi.

[3] M. Muntean, "Agile BI – The Future of BI,"

Informatica Economică, vol. volume 3, pp. 114-

124, 09/30 2013, doi: 10.12948/ISSN.

[4] D. Larson and V. Chang, "A review and future

direction of agile, business intelligence,

analytics and data science," International

Journal of Information Management, vol. 36,

no. 5, pp. 700-710, 2016, doi:

10.1016/j.ijinfomgt.2016.04.013.

Figure 4. Consider uncertainties in an agile approach

Figure 5. The role of end user in the development cycles in the agile

BI project


[5] R. Hughes, Agile Data Warehousing Project

Management: Business Intelligence Systems

Using Scrum. Elsevier Science, 2012.

[6] P. Serrador and J. K. Pinto, "Does Agile work?

— A quantitative analysis of agile project

success," International Journal of Project

Management, vol. 33, no. 5, pp. 1040-1051,

2015, doi: 10.1016/j.ijproman.2015.01.006.

[7] K. Schwaber, Agile Project Management with

Scrum. Sebastopol: Sebastopol : : Microsoft

Press, 2009.

[8] C. Larman and B. Vodde, Large-scale scrum :

more with less (Large-scale scrum). Boston:

Boston : : Addison-Wesley, 2017.

[9] S. W. Ambler and M. Lines, Disciplined agile

delivery : a practitioner's guide to agile software

delivery in the enterprise (Disciplined agile

delivery). Upper Saddle River, N.J.: Upper

Saddle River, N.J. : : IBM Press, 2012.

[10] R. Knaster and D. Leffingwell, SAFe 4.5

distilled : applying the Scaled Agile Framework

for Lean enterprises (SAFe 4.5 distilled).

Boston: Boston : : Addison-Wesley, 2019.

[11] W. Yeoh and A. Koronios, "Critical Success

Factors for Business Intelligence Systems,"

Journal of Computer Information Systems, vol.

50, no. 3, pp. 23-32, 2010, doi:

10.1080/08874417.2010.11645404.

[12] K. Collier, Agile analytics : a value-driven

approach to business intelligence and data

warehousing (Agile analytics). [Place of

publication not identified]: [Place of publication

not identified] : Addison Wesley, 2012, pp. 1

online resource (xxxv, 329 p.) :, ill.

[13] D. Linstedt and M. Olschimke, Building a

Scalable Data Warehouse with Data Vault 2.0.

Morgan Kaufmann Publishers Inc., 2015.

[14] R. Kimball and M. Ross, The data warehouse

toolkit : the complete guide to dimensional

modeling, 2nd ed. ed. New York: New York : :

Wiley, 2002.

[15] I. Bogdan-Andrei and P. Sorina, "Business

Intelligence. A Presentation of the Current Lead

Solutions and a Comparative Analysis of the

Main Providers," Database Systems Journal,

vol. V, no. 2, pp. 60-69, 2014.

[16] R. A. Mungree D., Morien D., "A Framework

for Understanding the Critical Success Factors

of Enterprise Business Intelligence

Implementation," presented at the AMCIS 2013,

2013.

[17] M. Stoica, M. Mircea, and B. Ghilic-Micu,

"Software Development: Agile vs. Traditional,"

Informatica Economica, vol. 17, no. 4, pp. 64-

76, 2013, doi:

10.12948/issn14531305/17.4.2013.06.

[18] K. Beck et al., "Manifesto for Agile Software

Development," in Manifesto for Agile Software

Development, ed, 2001.

[19] M. Cohn, User stories applied : for agile

software development (Addison-Wesley

signature series User stories applied). Boston:

Boston : : Addison-Wesley, 2004.

[20] I. Inayat, S. S. Salim, S. Marczak, M. Daneva,

and S. Shamshirband, "A systematic literature

review on agile requirements engineering

practices and challenges," Computers in Human

Behavior, vol. 51, pp. 915-929, 2015, doi:

10.1016/j.chb.2014.10.046.

[21] S. Ambler, "Adapting Agile Methods for

Complex The Agile Scaling Model (ASM):

Adapting Agile Methods for Complex

Environments," 01/11 2009.


towards an agile framework for business intelligence...

Documents