Habitual thinkingPatterns for IT disappointment and failure

Henrik Wivel & Jeppe Cramon

Corporate growth slowed dueto lack of funding

Berlingske Finans

6 out of 10 public projects going over time and budget

Version2

How well is your company doing?

• Are your IT projects delivered on time, on budget and with the right functionality?

• Are your IT systems running smoothly and stable?

• Does your system landscape contain more applications performing the same or similar tasks?

• Do you Outsource / Offshore the right tasks (or are you doing the right tasks in-house)?

How well is your company doing?

• Are your IT projects aligned with business needs?

• Is the business actively involved in IT projects and investments?

• Do you balance the entire organization's needs with the individual business needs in terms of architecture and infrastructure?

Does this seem familiar?

Each new project starts from scratch

PRODUCT VIEW

We need a combined customer view

Implement Siebel CRM in the Call center

We need to better manage our requirements

Buy and install HP Quality Center

We need a Service Orienteret Architecture

Build a solution using Web services in Oracle Service Bus / SeeBeyond / ...

SOA

Web Services

Requirement Management tools

BPM

N

Layere

d A

rch

itect

ure

Web

Sp

here

Work

flow

syst

em

Sh

are

Poin

t

Oracle Service bus

Team Foundation

Server

Sub

Vers

ion

Documentum

Portal

Ora

cle

DB

MS

SQ

L S

erv

er

NoS

QL

Policy Manager

WebLogic

.NET

JAVA

Ruby on Rails

iOS

Android

XML

JavaScript

Flash

BlackBerryGit

Hg

Dart

Cloud

Spring

Biz

Talk

Gro

ovy

Sca

la

F#C

#V

B.N

ET

REST

Typical management thinking

• We choose the tools and architecture that Gartner recommends

• More resources on a project = The sooner we will be finished

• 100 foreign developers are better than 10 local, even if the total price is the same

The Silver Bullet

All this leads to

• High complexity and a fragmented system landscape

• Strong need for product specialists who can not always participate across projects

• Danger of hiring habitual thinkers rather than technical domain experts

This creates the need for many resources on projects and therefore result in high costs

Same procedure as last year?

Same procedure as every year!!!

Because!!!

• “We haven’t got time to look at new technologies”

• “It requires more money and time”• “We have tried it before and failed”• “We realize we have challenges”• ”We _______________________________”

(Insert an excuse)

Open for new ideas?

Poul, you idiot, what's that!?You should only bring things we

can eat!

If it takes 1 man 10 days to dig atelephone cable , how many men do I needif I want it done in 2 days?

5 men in Denmark

10 men in Indien. Half time and even half price

How about using: Wireless network?

Do you still beat up your wife?

This is how we often ask

• Where can I get 200 developers for my project?• Where can I get them cheapest?• How do I scale my technical platform to 200

developers?• How do I manage large teams over a long

project cycle?

This is how we should ask

• Why do I need 200 developers?• How would I do if I "only" had:– 30 developers– 3 months and not 3 years for the project

• Do I need distributed teams?

How?What?

Why?

Is the solution Offshoring?

• 10 techinical domain experts in Denmark at $150 an hour = $1500 an hour

• 100 developer in China/India at $30 an hour = $3000 an hour

It’s a bit like bying app on the Apple AppStoreThey only cost $1

Project Economy

Productivity

Administrative overhead

Productivity

Administrative overhead

10 technical domain experts100 developers

Desperation

Inspiration

Desperation

SolutionsPattern for IT disappointment and failures

Here are the solutions!

SCRUMXP

LEANAgile

Outsourcing

What was it exactly we were trying to solve?

Getting closer to the business

Status Quo

We need a strong foundation for developmentAl

ignm

ent

Hig

hly

alig

ned

Less

alig

ned

EffciencyLess effective Highly effective

”Maintenance Zone” ”Well-Oiled IT”

”Alignment Trap”

11%

74%

7%

8%

% of the 504 respondents

+13

-14

-2 -15

+11

-6

+35

”IT enabled Growth”

% difference compared to the overall averages IT spendingCombined yearly growth-rate over a 3 year period

Sour

ce: B

ain

Anal

ysis

+0

A strong foundation is!

• Simple architecture• Assignments and data owned by one system• Automation• Tests

NOT a Simple architectureClient

Remote Facade

Application Service

Data Access Layer

Data Storage

Domain Object

DomainObject

Data Storage

Data Storage

Data Storage

DataService

DataService

DataService

ActivityService

ActivityService

Proces Service Proces Service

Client Client

Client

DataService

This does’t help either

A big unified model is every architects pibe dream

But as they say

ONE MODEL TO RULE THEM ALL

ONE MODEL TO FIND THEM

ONE MODEL TO BRING THEM ALL

AND IN THE DARKNESS BIND THEM

Still too big a model

Customer

Online Ordering System

Pricing

Inventory

Sales

Product

Unit PricePromotional PricePromotion End Date

Stock Keeping Unit (SKU)Quantity On Hand (QOH)Location Code

PriceQuantity Ordered

Description

There’s always more than one model at play in any big project

But when code based on multiple models is combined

… then the code becomes filled with errors, unreliable and difficult to understand

Communication between team members becomes confusing.

It is often unclear in what context a model should NOT be used!

Context is lacking

Smaller models and ”service reuse”Enterprise

Web ShopProduct

SKUDescriptionPriceQuantity Ordered… Inventory Service (SAP)

ProductSKUDescriptionQOHLocation Code…

Pricing ServiceProduct

SKUUnit PricePromotional Price…

Inventory

Pricing

Sales

Customers

Now even better – with an ESBEnterprise

Web ShopProduct

SKUDescriptionPriceQuantity Ordered… Inventory Service (SAP)

ProductSKUDescriptionQOHLocation Code…

Pricing ServiceProduct

SKUUnit PricePromotional Price…

Inventory

Pricing

Sales

Customers

Enterprise Service Bus

New and

”improved”

recipyWatch out!It's so easy to see a

product as the solution to all your

problems!

Before we had 1 problem.With the ESB we now have 2 problemes

Business focus is still lacking

Processes and state changes are important components

A much better recipyEnterprise

Web ShopProduct

SKUDescriptionPriceQuantity Ordered… Inventory Service (SAP)

ProductSKUDescriptionQOHLocation Code…

Pricing ServiceProduct

SKUUnit PricePromotional Price…

Inventory

Pricing

Sales

Customers

New SKU Event

New SKU Event

New SKU Event

New SKU Price EventNew SKU

Price Event

Mes

sage

Bus

What is a strong foundation?

• Simple architecture• Assignments and data owned by one system• Automation• Tests

In brief…

• Clean up the system landscape ...• Remove redundant systems• Remove systems that no longer have a reasonable

cost-benefit• Modernize systems which can still be make a

difference• Build new systems that can differentiate you from the

rest• ... And do it based on Neil Nickolaisen's Purpose

Alignment Model

Neil Nickolaisen’s Purpose Alignment Model

CriticalityLow High

Who cares? Keep on a par withcompetitors

PartnershipInvest

andDifferentiate

Market Differentiation

Grade

High

Low

What is a strong foundation?

• Simple architecture• Assignments and data owned by one system• Automation• Tests

Automation

• Continuous Integration– The code is automatically built and necessary tests

are conducted for verification• Setting up environments– Server setup, firewalls, deployment

• Integration testing• there is much more to get here ...

The solution in ETL

Electronic Landregistration

Background Information on Electronic Landregistration for those who have not heard about it ...

ETL – Suggested Architecture

Database

DataService

DataService

DataService

ActivityService

ActivityService

Proces Service Proces Service

Internal Portal External Portal

Integrations Service

IntegrationsService

Ora

cle

Serv

ice

Bus

WebLogic Portal

Suggested ETL Architecture

• Advantages– Logically a reasonable decoupiling (Single Responsibility

Principle)– Oracle makes good money on it

• Disadvantages– Logical layers are turned into physical layer– High complexity– Difficult to find resources that can work with the OSB– Stability– Performance (both runtime and for developers)– Deployment

ETL – Simplified Architecture

Database

Internal Portal External Portal

Integrations Service

IntegrationsService

Web

Logi

c Se

rver

Data Access Layer

Rich domain object Rich domain object

Business Service Business Service

Portal Facade

Java applications

Simplified ETL Architecture

• Advantages– Physical and logical architecture are closer to each other– Low complexity– Stability– Easier to find resources that can code Java– Performance (both runtime and for developers)– Deployment

• Disadvantages– The CV will not be boosted with Enterprise Service Bus– Oracle do not earn nearly as well ...

ETL - Automation

• Extremely complex domain area• The rules are determined by Law (not logic)• Covering registrations there are several

hundred years old• Complex logic required to automate law• Very short deadline for the third attempt to

build ETL

Context

Electronic Landregistration

• Technical choices– Programming language: Java– Database: Oracle

• First challenge– How do we integrate Java and Oracle database?

How it’s usually done

Java code

SQL

Means we go from this…

package dk.tigerteam.mdsd.demo.model.internal;

@Entity@Table(name = "Customer")public class Customer extends AbstractEntity { private static final long serialVersionUID = 2098912667L;

@Basic @Column(name = "name", nullable = false) private String name;

@OneToOne(cascade = { CascadeType.MERGE, CascadeType.PERSIST, CascadeType.REFRESH}, fetch = FetchType.LAZY) @JoinColumn(name = "addressId") @NotNull private dk.tigerteam.mdsd.demo.model.internal.Address address;

@OneToMany(cascade = { CascadeType.MERGE, CascadeType.PERSIST, CascadeType.REFRESH}, targetEntity = Booking.class, mappedBy = "customer", fetch = FetchType.LAZY) private Set<Booking> bookingCollection = new java.util.HashSet<Booking>();

public String getName() { return name; }

public void setName(String name) { this.name = name; } … … … … … … … … … …

}

package dk.tigerteam.mdsd.demo.mode.internal; @Entity@Table(name = "Booking")public class Booking extends AbstractEntity { private static final long serialVersionUID = 170080605L;

@Basic @Column(name = "comment", nullable = false) private String comment;

@Basic @Temporal(TemporalType.TIMESTAMP) @Column(name = "time", nullable = false) private java.util.Date time;

@Basic @Column(name = "timeslot", nullable = false) private int timeslot;

@ManyToOne(cascade = { CascadeType.MERGE, CascadeType.PERSIST, CascadeType.REFRESH}, fetch = FetchType.LAZY) @JoinColumn(nullable = false, name = "customerId") private Customer customer;

public String getComment() { return comment; }

public void setComment(String parameter) { this.comment = parameter; }

public java.util.Date getTime() { return time; } … … … … … … …

}

@Entity@Table(name = "Address")public class Address extends AbstractEntity { private static final long serialVersionUID = 1697028161L;

@Basic @Column(name = "street", nullable = false) private String street;

@Basic @Column(name = "zipCode", nullable = false) private String zipCode;

@Basic @Column(name = "city", nullable = false) private String city;

public String getStreet() { return street; }

public void setStreet(String parameter) { this.street = parameter; } … … …}

To this…

The line between modeling og muddling in hand written code is very thin

Can’t see the shoe for the toes?package dk.tigerteam.mddexample.model.customer;

@javax.persistence.Entitypublic class Customer extends dk.tigerteam.mddexample.model.user.User { private static final long serialVersionUID = 1328488396L;

@javax.persistence.Lob @javax.persistence.Basic @javax.persistence.Column(name = "comment") private String comment;

@javax.persistence.Embedded @javax.persistence.AttributeOverrides({@javax.persistence.AttributeOverride(name = "firstName",column = @javax.persistence.Column(name = "name_firstName") ) , @javax.persistence.AttributeOverride(name = "lastName",column = @javax.persistence.Column(name = "name_lastName") ) }) private dk.tigerteam.mddexample.model.customer.Name name;

@javax.persistence.Embedded @javax.persistence.AttributeOverrides({@javax.persistence.AttributeOverride(name = "street", column = @javax.persistence.Column(name = "address_street") ) , @javax.persistence.AttributeOverride(name = "state",column = @javax.persistence.Column(name = "address_state") ) , @javax.persistence.AttributeOverride(name = "zipCode",column = @javax.persistence.Column(name = "address_zipCode") ) , @javax.persistence.AttributeOverride(name = "city",column = @javax.persistence.Column(name = "address_city") ) , @javax.persistence.AttributeOverride(name = "country",column = @javax.persistence.Column(name = "address_country") ) }) private dk.tigerteam.mddexample.model.customer.Address address;

@javax.persistence.OneToMany(cascade = { javax.persistence.CascadeType.MERGE, javax.persistence.CascadeType.PERSIST, javax.persistence.CascadeType.REFRESH} , targetEntity = dk.tigerteam.mddexample.model.customer.Pet.class, mappedBy = "customer", fetch = javax.persistence.FetchType.LAZY) private java.util.Set<dk.tigerteam.mddexample.model.customer.Pet> petCollection = new java.util.HashSet<dk.tigerteam.mddexample.model.customer.Pet>();

@javax.persistence.OneToMany(cascade = { javax.persistence.CascadeType.MERGE, javax.persistence.CascadeType.PERSIST, javax.persistence.CascadeType.REFRESH} , fetch = javax.persistence.FetchType.LAZY) @javax.persistence.JoinTable(name = "Customer_invoice", joinColumns = @javax.persistence.JoinColumn(name = "CustomerId") , inverseJoinColumns = @javax.persistence.JoinColumn(name = "invoiceId") ) private java.util.Set<dk.tigerteam.mddexample.model.invoice.Invoice> invoiceCollection = new java.util.HashSet<dk.tigerteam.mddexample.model.invoice.Invoice>();

@javax.persistence.OneToMany(cascade = { javax.persistence.CascadeType.MERGE, javax.persistence.CascadeType.PERSIST, javax.persistence.CascadeType.REFRESH} , fetch = javax.persistence.FetchType.LAZY) @javax.persistence.JoinTable(name = "Customer_appointment", joinColumns = @javax.persistence.JoinColumn(name = "CustomerId") , inverseJoinColumns = @javax.persistence.JoinColumn(name = "appointmentId") ) private java.util.Set<dk.tigerteam.mddexample.model.customer.Appointment> appointmentCollection = new java.util.HashSet<dk.tigerteam.mddexample.model.customer.Appointment> ();

public String getComment() { return comment; }

public void setComment(String parameter) { this.comment = parameter; }

public dk.tigerteam.mddexample.model.customer.Name getName() { return name; }

public void setName(dk.tigerteam.mddexample.model.customer.Name parameter) { this.name = parameter; }

public dk.tigerteam.mddexample.model.customer.Address getAddress() { return address; }

public void setAddress( dk.tigerteam.mddexample.model.customer.Address parameter) { this.address = parameter; }

public java.util.Set<dk.tigerteam.mddexample.model.customer.Pet> getPetCollection() { return petCollection; }

public java.util.Set<dk.tigerteam.mddexample.model.invoice.Invoice> getInvoiceCollection() { return invoiceCollection; }

public void setInvoiceCollection( java.util.Set<dk.tigerteam.mddexample.model.invoice.Invoice> parameter) { this.invoiceCollection = parameter; }

public java.util.Set<dk.tigerteam.mddexample.model.customer.Appointment> getAppointmentCollection() { return appointmentCollection; }

public void setAppointmentCollection( java.util.Set<dk.tigerteam.mddexample.model.customer.Appointment> parameter) { this.appointmentCollection = parameter; }}

When all you wanted to convey was this

Hand held consistency…

@Entitypublic class Customer { @OneToMany( targetEntity = Pet.class, mappedBy = "customer” ) private Set<Pet> petCollection = new HashSet<Pet>();

public Set<Pet> getPetCollection() { return new OneToManySetWrapper<Customer, Pet>(this, petCollection) { @Override protected Customer getOneSideObjectInManySideObject(Pet manySideObject) { return manySideObject.getCustomer(); }

@Override protected void setOneSideObjectInManySideObject(Pet manySideObject, Customer oneSideObject) { manySideObject.setCustomer(oneSideObject); } }; }}

@Entitypublic class Pet { @ManyToOne private Customer customer;

public void setCustomer(Customer parameter) { new ManyToOneWrapper<Customer, Pet>(this) { @Override protected void addManySideObjectToOneSideCollection(Customer oneSide, Pet manySide) { ((WrappedSet<Pet>) oneSide.getPetCollection()).getWrappedCollection().add(manySide); } @Override protected void removeManySideObjectFromOneSideCollection(Customer oneSide, Pet manySide) { ((WrappedSet<Pet>) oneSide.getPetCollection()).getWrappedCollection().remove(manySide); }

@Override protected Customer getOneSideObjectInManySideObject(Pet manySide) { return manySide.customer; }

@Override protected void setOneSideObjectInManySideObject(Pet manySide,Customer oneSide) { manySide.customer = oneSide; } }.updateOneSideObject(parameter); }}

Automation

From sweatshops to automation

At ETL we introduced this processUML class diagrams

Java codeHibernate

SQL

WSDLTests

Agile process

Another possibility

Build your own language

Sounds hard, but it’s quite easy Ru

le L

angu

age

Met

a M

odel

Rule Grammar (using Xtext)

+

Dat

a La

ngua

ge M

eta

Mod

el

+

Editor & IDE

Text Editor

Point of No Return, traditionally

Time

Cost of Rework

V 1.0

Hacks!

Freeze

Point of No Return, Model Driven

Time

Cost of Rework

V 1.0

Hacks!

Freeze

The end of writing tedious code by hand

Frameworks

(Hibernate/JPA, Entity

Framework)

Libraries (f.eks. Java/.NET)

Schematic code

Interesting code

(Written by hand)

Written by hand

Model Generator

What do we get out of it?

• Higher level of abstraction• Technical decisions can be postponed• "Point of No Return" can be postponed• Documentation is always update to date• Higher Quality and Uniformity• Refactoring is both faster and cheaper• Short development cycle

The real short version

We can achieve the same with fewer resources

We can achieve the same in less time

We can achieve more with the the sameresources and at the same time

Electronic Land registration

Questions?@TigerTeamDK on Twitterhttp://tigerteam.dk