emergent architecture- a casestudy treds

Copyright 2010 Code71 Inc. All rights reserved.http://www.code71.com http://www.scrumpad.com

Emergent Architecture on Agile Projects

Syed Rayhan

Co-founder, Code71, Inc.Contact: [email protected]: http://blog.syedrayhan.comCompany: http://www.code71.comProduct: http://www.scrumpad.com


2

Agenda

Recap

Initial Architecture

Section 2

Section 4

Section 3

Evolving initial architecture

Project summary

Section 5

IntroductionSection 1

Q&ASection 6


3

Agenda

Introduction


4

My Background

Expertise

Career

Iterative incremental development

Agile technology planning and architecture

On-shore/Off-shore software development using Agile/Scrum

Interests

Co-founder, Code71, Inc. 15+ years of total experience Co-author of “Enterprise Java with UML”

Cultural aspect of self-organizing team Scrum for small projects delivered remotely Agile engineering practices


5

What to Expect

Focus

Context

How architecture fits in the new, new world of agile development

How emergent architecture actually is in practice

Candid look at what is working, what is not working, and how we have adjusted to succeed

Key Takeaways

How to do architecture on agile projects

Who does architecture

What practices to customize and why

Iterative, incremental development a.k.a. Agile software development

Traditional architecture practice is inherently waterfall


6

The Case Study• Department of Motor Vehicle (custodian agency)

• Department of Transportation

• State and local law enforcement agencies

ProjectJustification

Clients

Architecture Landscape

• Ensure public safety on roads

• Reduce traffic collisions

• System under consideration is dependent on other systems that are also in the process of being replaced

• Infrastructure is provided by VITA

• New technology


7

Agenda

Project Summary


8

Problem Statement and Project Goals

• The current backlog of reports 3-6 months• Duplicate data stores is causing poor data quality• Manual data entry is causing poor data quality as well as

increased processing time and costs• Lack of data visibility and access to data contributes to

wrong or delayed policy changes and or enforcement

Problems

Goals

• Streamlining and simplification of data collection• Elimination of paper reports• Increasing the efficiency and improving data quality• Elimination of duplicate data entry• Electronic data reporting• Elimination of data entry backlogs


9

Project Facts

• Contractors and consultants

• Core team has 12 members with cross-functional

• Located on-site

Funding

Staffing

Timeline

• Federal grant

• $4.5 M

• Project initiated in 2006

• Development started in 2007 May

• 60 sprints completed to-date

• First release was in March 2008


10

Project Context


11

Project type

well understood Not in agreement

expert

new

requirements

tech

nolo

gy

simple

Zone of death

complex

complex

Our project


12

Existing Architecture

data entry

Image db 2

Image db 1

data entry

copy

update

image

image

update

update

Agency 2Agency 1

Agency 3

data export

Main App

App 1 App 2

App 3


13

Existing Enterprise Architecture Standards

Technology Platform

Security

Infrastructure

VB.Net, MSSQL & Oracle

Two-factor authentication (FOB based)

Strict 3-tier architecture


14

Agenda

Initial Architecture


15

Architecture Principles

• Minimize custom components

• Reuse existing components

• Leverage COTS components


16

Needs to Technology Map

Crash form redesign

Electronic submission

Increase dataaccuracy and reducedata correction needs

Workflow automation

Information exchangewith various externaland internal systems

Standard & ad-hocReporting

Analysis and datamining

Authentication &authorization(security)

Expanded and modifieddata model

Support for multiple waysto send in reports (scan,Web, batch)

Automated data validation

Workflow Management

Integration tool that cansupport multipleformats, protocols, andmodes (batch & real-time)

Reporting tool forbusiness & technicalusersAnalytic tool forBusiness users

Security that spansIntranet, Extranet, &Internet

XML

Report Beam,

Taskmaster

BizTalk

BRE

WF

SQL Server

SSIS, SSRS

LDAP based Store (ADAM)

Functional Technical Proposed

Requirement Assessment Component


17

Architecture Patterns

TREDS User Interface

TREDS business logic controller

CRASH Data Model

TREDS Integration Layer

HTRIS RNS

CSSCAP

RMSRMS

Report Beam

MVC

Hub & Spoke


18

Logical Architecture- Initial Vision


19

Roadmap (partial view)

Nov. 15, 2007 - CAP Expansion

•CAP expansion to accommodate new fields

•Scan bubble data into CAP•Automate data presentation forSafetynet•Build TREDS/SFTP Interface•Build system to system authorization and authentication

Milestones Capabilities Impact

Jan. 15, 2008•Revised FR300 Scanning•TREDS/Safetynet Integration

•Allow automated commercial datareporting to Safetynet•Reduce manual data entry with scanning•Business Process Impact:: - Scan data validation - Automation of some current manual operations

•Expand database for new fields•Expand UI for new field data entry

April 15, 2008- TREDS/RNS Integration

•Build TREDS/CAP interface•Build TREDS/RNS interface•Build CAP/Broker interface

•Eliminate RNS dependency on HTRIS•Remove requirement for FR300forwarding to VDOT•Business Process Impact: - DMV/VDOTFR300 forwarding process

Nov. 15, 2008•TREDS Core Deployment•TREDS Client – Pilot 1

•Build TREDS data management•Build centralized rules engine•Build TREDS Data Warehouse(Havana)•Scan bubble data into TREDS(CUBA)•CAP data migration to TREDS•Build human to system authorization and authentication

•Back Office operations will use

TREDS for data management

•TREDS/ CAP data synchronization will provide system fall back during burn in•Policy and standards enhancements may occur due to human to system authentication and authorization•Business process impact – IT systems support for TREDS – operations will expand


20

Agenda

Evolution of Architecture


21

1st Release Architecture

Controller WS

BT Orchestration

WS

Broker

FTP

data migration

App 1

Main App

unprocessed data

update

WS-API

Image db1image

Scanning App

Image db2

App 3

image

data

DW

data

image

App 2

scannedreport


22

1st Release Recap

Reduce data entry time by 50%

Reduce architectural risk around infrastructure,

Existing system is still the system of records

Electronic data capture, not delivery

Goals

Capabilities

Lessons Learned

Form redesign should have followed data modeling and scanning technology selection

Windows environment for efficient data entry poses different challenge

Result

OCR accuracy was lower than expected

Initial data processing speed went down


23

2nd Release Architecture

Controller WS

BT Orchestration

WS

Broker

FTPApp 1

Main App

data

update

WS-APIImage db1

image

Scanning App

Image db2

App 3

image

data

DWprocessed

data

App 2Workflow

Model

Web Client

DB

BRE

DB

AzManscannedreport

Processed data

AuditDB

trigger


24

Defining Workflow using State Machine

stm Crash State Diagram

PO submits the original/revised report

Superv isor rev iew

Under Correction

Pending

Under Rev iew On hold

DMV Approv ed

Assigned Pending

Anoriginal/revisioncrash wasscanned

Pending Manager Rev iew

Accepted

BO manager/clerkreaccess crash

An approved crash is submitted

PS rejects crashto PO

PS approves crash

PO submits for re-approval

system assignsreportables or BOclerk pulls

Non-reportable by pass the crash processing

BO manager/clerkputs in on hold

BO clerk sends crash to manager

A new revision created

Manager reassigns crashto pending or a user

VDOT accepts crash with updated GPS

VDOT rejects the crash

BO clerk processes crash

Time out or logoff

BO manager processes crash

Crash lifecycle ends

Manager/clerk accepts crash

Auto-approved

Revision is created


25

2nd Release Recap

Existing system is still the system of records

Automated work management of data entry clerks

Goals

Capabilities

Lessons Learned

Too many data validation rules were too restrictive

Lack of robust exception handling resulted in premature termination of workflows

Result

Automated workflow was well received

Improved the quality of management of the supervisors

Introduce automated workflow for data processing


26

3rd Release Architecture

Controller WS

BT Orchestration

WS

Broker

FTP

App 2

Main App

update

WS-APIImage db1

image

Scanning App

Image db2

App 4

image

data

DW processed data

App 3Workflow

Model

Web Client

DB

BRE

DB

AzMan

Report WS

scannedreport

AuditDB

trigger


27

3rd Release Recap

Integration with App2

Canned reporting from new DW

Main reporting from the new system

Complete data entry from the new system

Goals

Capabilities

Lessons Learned

Longer processing time at certain steps in the workflow caused user frustration

Web service interface to BizTalk was causing timeouts

Result

Reporting time reduced from a few weeks to a few minutes

Reports requiring manual check reduced to 70%-80% due to auto acceptance


28

4th Release Architecture

Controller WS

BT Orchestration

WS

Broker

FTP

App 2

Main App

update

WS-APIImage db1

image

Scanning App

scannedreport

Image db2

App 4

image

data

DWprocessed

data

App 3Workflow

Model

Web Client

DB

BRE

DB

AzMan

Report WS

Req Q Req Q

Client App

electronicreport

scanner

AuditDB


29

Workflow with Asynch. System Processing

stm Crash State Diagram

PO submits the original/revised report

Superv isor rev iew

Under Correction

Pending

Under Rev iew On hold

DMV Approv ed

Assigned Pending

Anoriginal/revisioncrash wasscanned

Pending Manager Rev iew

Accepted

System Processing

BO manager/clerkreaccess crash

An approved crash is submitted

PS rejects crashto PO

PS approves crash

PO submits for re-approval

system assignsreportables or BOclerk pulls

Non-reportable by pass the crash processing

Auto-approved

BO manager/clerkputs in on hold

BO clerk sends crash to manager

A new revision created

Manager reassigns crashto pending or a user

VDOT accepts crash with updated GPS

VDOT rejects the crash

Revision is created

BO clerk processes crash

Time out or logoff

BO manager processes crash

Crash lifecycle ends

BO manager/clerk accepts the crashCSS post errors


30

4th Release Recap

Improve response time using asynchronous processing

Electronic delivery of reports from the field

Direct submission of reports (no scanning) to the system

Complete data entry from the new system

Goals

Capabilities

Lessons Learned

Change in workflow caused versionites

Accelerate adoption of electronic delivery of reports from the field

Result

Improved user satisfaction from reduction in response time

Reports requiring manual check further reduced to 50%-60% due to auto acceptance


31

5th Release Architecture

Controller WS

BT Orchestration

WS

Broker

FTP

App 2

Main App

update

WS-APIImage db1

image

Scanning App

Image db2

App 4

image

data

DW processed data

App 3

Workflow

Model

Web Client

DB

BRE

DB

AzMan

Report WS

Req Q Req Q

Client App

electronicreport

scanner

scannedreport

AuditDB

WCF


32

5th Release Recap

Integration with App3“Revision of Reports” handlingAd-hoc reporting from new DW

New system is self-sufficient

Asynchronous processing of reports delivered electronically

Goals

Capabilities

Lessons Learned

The definition of “revision of a report” changed (new capability changes business practices)

ResultIntegration with other reporting tools for electronic reporting

Old system is ready for sunset


33

What’s next?

• Keep up with the technology upgrades and changes• Upgrade to Biztalk 2009• Upgrade to Visual Studio 2008

• Keep an eye on new technology for new capability or easier maintainability• ASP.Net MVC• Entity Framework or Fluent NHibernate to replace Nhibernate


34

Agenda

Recap


35

Recap lessons learned

Start with existing architecture standards, identify gaps, and know

upcoming changes

Influence / introduce new standards to fill the gap in existing

architecture

Align with upcoming changes to the existing architecture

Let architecture evolve top-down (architect is responsible) and

bottom-up (team is responsible)

Use working “spikes” to determine how to adopt new technologies

Manage evolution of architecture in small increments (a.k.a refactor)

Document architecture using FAQ style and just-in-time

Architecture evolves; when unmanaged, it becomes sprawling.


36

Q&A

Guided evolution of architecture improves Agility!


37

Contact

Please contact for on-site

training/coaching or Webinar:

Contact: [email protected]: http://blog.syedrayhan.comCompany: http://www.code71.comProduct: http://www.scrumpad.com

emergent architecture- a casestudy treds

Documents

emergent architecture

agile software development

architecture fits

new world of agile development

agile technology planning

small projects

coauthor of enterprise

succeedfocuskey takeaways