Luis [email protected]
uk.linkedin.com/in/lweir@luisw19
soa4u.co.uk/
A MICROSERVICE
APPROACH
FOR
LEGACY
MODERNISATION
Beyond The HorizonJune 2,3 2016
{2}
About me
Luis Weir Oracle Ace Director – Principal Architect
in assisting organisations define and implement solutions and strategies that can helpthem realise the benefits that such technologies have to offer.I am very passionate about technology. I have be the lead authored of two books (OracleSOA Governance 11g Implementation and Oracle API Management 12c Implementation), I am aregular blogger and speaker in major conferences and events. A well-known industry expertespecially when it comes to Oracle middleware technologies I am also an OTN certified SOAblack belt.
I am an Oracle Ace Director, principal architect and a thought leaderspecialised in Oracle Fusion Middleware & Oracle PaaS technologies. Withmore than 15 years of experience implementing IT solutions across theglobe, I have been exposed to a wide wide variety of business problemsmany of which I’ve helped solved by adopting SOA architectural styles suchas traditional SOA, API management and now Microservices. My current focusis
2nd Place1st OTN Cloud HackathonJune, 2016
Cloud Contribution AwardSOA CommunityMarch, 2016
Latest Media:•Oracle Magazine May/June 2016 (http://bit.ly/1RTCAU3)•Systematic Approach for Migrating to Oracle Cloud SaaS(http://bit.ly/1Xr6acs)•Oracle Magazine Jan/Feb 2016 (http://ora.cl/Vhh)•API Management Implementation (http://ora.cl/Gcw)•A Word About Microservices and SOA (http://bit.ly/25Dk5go)
{4}
Agenda
• Modernisation strategies & approach
• Microservice architecture
• Seven conclusions
• Context, driving forces and challenges
{5}
… what is a legacy system?
Monolithic applications that significantly resists modification and evolution [1], seem frozen in time [6] and thus do not fit with the organisations future IT strategy [7]
Characteristics:
• They are business critical [1][2][4][5][6][7][8][12][15]
• Typically been around for a while (in some cases 30+ years) [2][7][8]
• End of life tech stacks [1][2][5][8][12][15]
• Not necessarily defined by age. Lack of vendor support or inflexibility also mean legacy [2][5][7][14]
• They are stable, reliable and performing [4][5][7][14]
{6}
Does the world run on legacy?• 80% of the worlds systems are legacy [3][5]• 200 billion lines of legacy code are still in use [5][7]• 1+ million COBOL programmers world wide maintaining an approximate of 100 billion lines of COBOL code [7][11]
• 75% of financial institutions in Europe are using out-dated systems (legacy) for core business [9]
• COBOL powers 80% of all daily business transactions [17]• 30+ years of legacy modernisation research and 80% of world systems are still “legacy”? [10][7]
[11]
{7}
Driving forces
• Lack of flexibility. Extremely difficult to change to take new requirements affecting new product development and time to market [1][2][3][4][5][7][8][9][12][15]
• High costs of maintenance and change [1][2][3][4][5][7][8][9][12][15]
• End of life tech-stacks(out of support, no new or major feature releases) [1][2][3][5][7][8][9][12][15]
• People [with relevant skills] are scarce[4][5][7][10][11][17]
• Integration is difficult [1][4][5][6][12][15]
• Security (no new security patches) [6][9][13]
{9}
Arguments against modernisation• High risk of failure – “we’ve tried before (several times) and failed” [1][2][4][6][7][10][12][15]
• Too costly without clear justification. Ie. When will we see ROI? [1][2][3][4][6][7][8][12][14][15]
• Lack of knowledge and skills –[1][4][5][7][12][10][11][14][17]
• “If ain’t broken don’t fix it”, the system is doing what’s supposed to why change it? [3][6][7][14]
“They [Top management] are always looking for a short term Return of Investment. Once they put the money in, they want to earn it back” [7]
We didn’t fix it last year, and survived. Why should this year be different? [7]
“I think top management doesn’t understand the issue and they don’t give budget for it [legacy modernisation]” [7]
{11}
Faults are as big as the legacy!!
(2012) 16 million (RBS, NatWest, Ulster Bank) customer accounts
ended up being frozen
(Dec, 2014) Cancelation of 40 flights in UK affecting 1.9k
other flights and 230k passengers
(Jan, 2016) Customers couldn't access accounts for 2 days
{12}
Not failing is not success [16]
Insight into legacy modernisation success rate:
• Success rates vary from 24% to 39%, depending on study and the type of technology project [16]
• 1 in 6 projects (17%) result in "black swan” [16]
• Only 50% of projects deliver 44% of the intended business benefits [16]
“Healthcare.gov is just a frontend. The heavy lifting takes place in the back ends, a extremely complex array of legacy systems that span multiple US federal agencies” [20]
“The worry is that trying to promise a startup-style experience on the back of multiple old and lumbering IT systems. There's a long way to go, even after all the initial issues with accessing the site itself are resolved” [20]
{13}
Usual suspects: reasons for failure• Lack of documentation and knowledge of the system leading to issues like bad scoping, poor estimation and planning, inappropriate target designs and incomplete testing [1][2][3][4][7][8][12][14]
• Wrong approach specially based on “Eating the elephant in one go” (cold turkey)” [1][4][7][8][12][15][16]
• Time pressures. Rushing through project phases to meet deadlines – [1][4][5][7][12][10][11][14][17]
• Lack of people with relevant skills –[1][4][5][7][12][10][11][14][17]
{14}
Agenda
• Modernisation strategies & approach
• Microservice architecture
• Seven conclusions
• Context, driving forces and challenges
{15}
What is a Microservice? SOA 2.0?
Loosely coupled service oriented architecture with bounded context [27]
“The value of the term microservices is that it allows to put a label on a useful subset of the SOA terminology” [22]
[21]
Functional decomposition of systems into manageable and
independently deployable components[18]
{17}
MSA – Technical
Microservice
Run on its own process
Deployed independently
Scales independently
Owns its data
Is stateless
Isolates faults
{18}
MSA - Technical
Legacy System Monolith
Mainframe (ie IBM System Z, S/360)
Storage(ie. DB2, IMS/DB)
Hypervisor (ie. PR/SM –Type 1)
Application Services(ie. CICS)
Hardware Resources(ie. DADB, IDMS,IEDN)
Operating System(ie. z/OS, z/VSE)
Batch Services(ie JCL,JES,3rd p.)
Modern Monolith
Relational Database
Hardware Resources
Host OS
Hypervisor (type 1 or 2)
Guest OS (VMs)
Bin/Libs(MREs, Interpreters, etc)
Application Server
Application(ie. ESB)
Services
Any Hardware
Container Engine
Microservice Architecture
Mongo
Cassandra
Oracle HBase
Neo4j
Hardware Resources
Host OS
Guest OS (VMs)
Scala Java
Ruby
Node
Jolie
Services
Any Hardware
Container Engine
Bin/Libs
Bin/Libs
Bin/Libs
Bin/Libs
Container Engine
Bin/LibsHypervisor
(type 1 or 2)
{19}
MSA – Architectural
Microservice
Bounded context
Single responsibility
Choreographed
Smart endpoint and dump pipe
Polyglot
API gateways
{20}
Order & FulfillmentDomain
Customer RelationsDomain
ERPDomain(P2C, R2C,HR, GL,Billing,etc)
MSA – Architectural
API Gateway(s)
Microservice Architecture
AsyncCommunication
Microservice Architecture Legacy
Message Pipe
Mobile Apps
Adapter
Adapter
SyncCommunication Managed API Microservice
MonolithServiceChoreography
Contact
Customer
BoundedContext
Shipment
Order Product
Web Apps Applications
{21}
MSA - ArchitecturalPattern Traditional SOA MSA
Monolith pattern (http://bit.ly/1Gjr2Y0) Yes No
Polyglot Programming & Persistence(http://bit.ly/18BvDIj & http://bit.ly/1XYiak2)
Not traditionally (use of Suites)
Yes
API gateway pattern (http://bit.ly/1WTyNLJ) Yes Yes
Orchestration (http://bit.ly/1U0SWil) Yes No
Choreography (http://bit.ly/1ssALZQ) No Yes
Event Collaboration (http://bit.ly/25Dk7oE) Yes Yes
Canonical Schema (http://bit.ly/1r6KkfK) Very common No
Schema centralization (http://bit.ly/1sVlqkc) Very common No
Decouple Contract (http://bit.ly/1O8mVpm) Yes Could be….
Bounded Context (http://bit.ly/1o7AK8B) Some times Yes
Ubiquitous Language (http://bit.ly/1c8nXQe) Some times Yes
Bulkhead (http://bit.ly/1c8nXQe) Not really… Yes
Tolerant Reader (http://bit.ly/1aa4mr9) Some times Yes
Client-side Service Discovery(http://bit.ly/1OunUyq)
Initially only(service registry)
Recommended
Server-side Service Discovery(http://bit.ly/1X3RmzA)
Yes Yes
ESB Pattern (http://bit.ly/1ZlSKeT) Yes Across bounded contexts or domains (dump pipe)
[19]
Yes = Applied most of the timeNo = Not applied most of the time
{22}
MSA – Organisational
Microservice
Teams organized around business capabilities
Small teams
You build it you run it
Decentralised governance
Culture of automation
Products not projects
{23}
MSA – Organizational
Development and support teams organized by technologies resulting in siloes(Conway’s law in action)
SOA
Support
Team
DB
Support
Team
UI
Support
team
UI Dev Team
Database Dev Teams
SOA Dev Team
Project Teams
ComsGaps
Multi-disciplinary [small] teams organized by business capability resulting in modular systems
CustomerUI D
B
MW
OrdersUI D
B
MW
ItemsUI D
B
MW
ShipmentUI D
B
MW
Traditional Operations Model
DevOps /Continuous Delivery
MSA Operations Model
[19]
CustomerMicroservice
OrdersMicroservice
ItemsMicroservice
ShipmentMicroservice
{24}
Recommend watching
https://www.youtube.com/watch?v=yPvef9R3k-M
DDD & MicroservicesBy Eric EvansGoto Berlin, Nov 2014
https://www.youtube.com/watch?v=nMTaS07i3jk
State of the art in MSABy Adrian CockcroftDockercon Amsterdam, Nov 2015
https://www.youtube.com/watch?v=wgdBVIX9ifA
MicroservicesBy Martin FowlerGoto Berlin, Nov 2014
{25}
Recommend watching (continuation)
http://bit.ly/1UBDQ2T
7 deadly sins of MicroservicesBy Daniel BryantJan 2014
https://www.youtube.com/watch?v=PFQnNFe27kU
Principles of MicroservicesBy Sam NewmanDevoxx Belgium, Nov 2015
{26}
Ok cool, but why MSA for this?
ModularityEat the elephant one piece at the time. Phased implementation approach. Starting small. Small teams owning full lifecycle of their piece(a business capability)[23]
Segmented complexitySeparate a big problem into smaller problems handled by small teams ensures mental models are retained avoiding a “legacy in the making” [5]
Easy of deployment / speedMoving away from entire system deployment (ie. “one line change to a million-line-long monolithic”). Deploy services independently and fast (ie. with containers) and introduce automation (continuous delivery) [23]
{27}
(continuation)
Scalability & resilienceScale independently and possibly on-demand. Bulkheads to isolate problems and avoid whole system failures (avoiding the cascade effect), Then purposely test resilience [23]
Enabling cloud transitionBuilding container-based modular applications whilst adhering to basic principles (like 12 factor [30], Lehman’s law [31], and the reactive manifesto [33]) cloud adoption is a real option
Breaking organisational silosOrganise small teams based on business capabilities in order to avoid organisational silos being reflected in the way systems are built (Conway's law [32])
{28}
Agenda
• Modernisation strategies & approach
• Microservice architecture
• Seven conclusions
• Context, driving forces and challenges
{29}
Modernisation Strategies
Cold Turkey [8]
• Big-bang approach• High risk and Costly (big project)• Reduced business benefits
Monolith
Ie. COBOL
Monolith
Ie. Java
Tool
Refactor / ReplaceConvert legacy code and/or replace with COTS
Monolith
Ie. COBOL
Tools, manual,
any other..
RebuildRebuild from ground up in modern language in one-go
Monolith
Ie. Java
Apps
Chicken Little [8]
• Phased approach based on coexisting with legacy using gateways• Less risky but complex
Monolith Decompose
A1
Gateways
An
Re-engineerDecompose legacy into modules and then rebuild whilst coexisting with legacy using gateways
Butterfly [15]
• Phased approach based on coexisting with legacy without gateways• Less risky but complexRe-engineerDecompose legacy into modules and then rebuild whilst coexisting with legacy using data access allocators and chrysaliser
Monolith
Decompose
A1
An
Chrysaliser
Wrapping[15] • As little modification to legacy as possible (tactical approach in nature)• Used of wrapper technologies like CICS, screen-scrapers or ESBs with legacy adapters• Lower risk, quicker results• Higher costs, complexity and risk in the mid-long term
Monolith
A1
Wrappers
An
{31}
I) API-fication (wrap)
OnpremiseTerminals
Oracle Cloud PaaS
MainframeDB2
OrderMng
Inventory
Pricing
Product
ShipmentConsignment
s
Sales Purchasing Payables
Receivables GL Etc..
WeblogicAPI Gateway (OAPCS)
Oracle SOA Suite(11g or 12c when Adapter is certified)
Product API
CICS PROGRAMS Oracle Connect
JCA Legacy Adapter
REST Adapter
Native
Apps
Web
Apps
OACCS MCSJet on Node Mobile Backend
{32}
Oracle Cloud PaaS
APICS Gateway
Native
Apps
Web
Apps
I) API-fication (wrap)
Terminals
MainframeDB2
OrderMng
Inventory
Pricing
Product
ShipmentConsignment
s
Sales Purchasing Payables
Receivables GL Etc..
Weblogic
Oracle SOA Suite(11g or 12c when Adapter is certified)
CICS PROGRAMSOracle Connect
JCA Legacy Adapter
REST Adapter
OACCS MCS
ICS
Jet on Node
Onpremise Agent
Mobile Backend
Onpremise
Rest endpoint
Product API
{33}
Weblogic
Oracle SOA Suite(11g or 12c when
Adapter is certified)
Oracle Cloud PaaS
APICS Gateway
Native
Apps
Web
Apps
II) Re-engineer (MSA) & coexist
Terminals
MainframeDB2
Inventory Orders
Pricing
Product
ShipmentConsignmen
ts
Sales Purchasing Payables
Receivables GL Etc..
CICS PROGRAMS
JCA Legacy Adapter
REST Adapter
OACCS MCS
ICS
Jet on Node
Onpremise Agent
Mobile Backend
Onpremise
Rest endpoint
OACCS ODCS
ProductProductProduct API
Oracle Connect
ODIDB2 KM
Sync
{34}
Weblogic
Oracle SOA Suite(11g or 12c when
Adapter is certified)
Oracle Cloud PaaS
APICS Gateway
Native
Apps
Web
Apps
III) Switch over & start over
Terminals
MainframeDB2
Inventory Orders
Pricing
Product
ShipmentConsignment
s
Sales Purchasing Payables
Receivables GL Etc..
CICS PROGRAMSOracle Connect
JCA Legacy Adapter
REST Adapter
OACCS MCS
ICS
Jet on Node
Onpremise Agent
Mobile Backend
Onpremise
Rest endpoint
OACCS ODCS
Product APIInventory API
ODIDB2 KM
updates
ProductProductProducts Changes in terminals blocked
{35}
Oracle PaaS Cloud
APICS
OACCS
Product
Inventory
Orders
Pricing
ShipmentConsigments
etc
Gateway(s)
OACCS Jet on Node MCSMobile Backend JCS ADF, etcOther CS’s
Sales
ODCS
ICS/SOACS
…
…
…
The goal…
Web
Apps
Native
Apps
Onpremise
MainframeDB2
xxx
{36}
Functional Decomposition
Activity
Output
Team
Business Process Flows (L1-3)
Logical Data Model
Business Rules
Document Analysis
Black-Box Analysis
SME Interviews
Process Modeling
Interface catalogues
Top-down requirement analysis
Business Analysts
Solution Architects
User Stories
DomainModel
Decomposition Approach
Technical Decomposition
Code Reverse Reengineering
Bottom-up requirement analysis
Reversed UML (L4-5)
Gap Analysis
Code Functionality Mapping
Software Engineers
Business Analysts
Solution Architects
Reversed Physical Data Model
Reversed Business Rules
Business Logic / Rules Discovery
Solution Design
Solution Design
Business Capability Model
Target State Solution Architecture
UX Patterns
Product Backlog & Sprint Planning
Build Planning
UX Designers
Business Analysts
Solution Architects
Product Manager
{37}
The WAGILE Approach
Sprint 1
Sprint 2
Sprint 3
Sprint 4
Sprint 5
Sprint N
W-Agile Methodology
Functional
Technical
Solution Design
New Requirements
DecompositionsEnd to end
testing
SIT, UAT, PPT
Release Support
Several iterations – small scopeFew iterations – large scope
Solution build & component test
Factory Management & Governance
Centre of Excellence
Release
Mgmt. &
Deployment
Detail Design
Sprint Backlog
Sprints
Scrum Master
DevOps
Pre-SITModernisation
Factory
Build & Test
Sprints
DevOps
Work Items Job Card
Cutover
Go-Live
Rollout
{38}
Agenda
• Modernisation strategies & approach
• Microservice architecture
• Seven conclusions
• Context, driving forces and challenges
{39}
I) Avoid the obvious!
Avoid legacy to legacy modernisation!!
A fundamental goal of legacy systems modernisation is that the target system
doesn’t become a legacy [8]. Modular systems are easier to evolve
{40}
II) No big bangs or cold turkeys
One piece a the time!
Understand the problem. Slice and dice your elephant by defining boundaries in the
business capabilities. Modernise one piece at the time. Starting small
Replenishment
Orders
SalesOrders
Logistics
Tracking
{41}
III)One domain to rule them all [26]
Gather together those things that change for the same reason, and separate those things that change for different reasons [23][25]
[24]
Domain driven design (DDD) divides up a large system into Bounded Contexts, each of which can have a unified model - essentially a way of structuring Multiple Canonical Models[24]
{42}
IV) The main prereq is setting the right goals
Do “You have to be this tall to use microservices[28]”? – perhaps not!
Sure you need a degree of operational maturity to adopt microservices, however
maturity is a journey that can be achieved by setting the right goals and objectives.
So set the right goals!!!
[27]
{43}
V) Optimise for speed not efficiency [29]
Speed means learning about your customers and giving them what they want at a faster
pace[29]. Use simple patterns automated by tooling [27]. Learn the walk before running.
Speed wins in the marketplace [27]
{44}
VI) 30+ years of research [5][10] proves that:
What you have seen early is just a point of view (mine!). Do dui-diligence and find the right approach for your environment. Get inspiration from lots of research work
There is no silver bullet [26]
{45}
VII) The world runs on legacy
So, let’s learn legacy!
Credible research shows that there already is a huge skill gap in the industry on COBOL, RPG, PL/I, FORTRAN, PASCAL, C++, etc [4][5][7][10][11][17].
“Study the past if you would define the future” -Confucius
{46}
References (I)[1] Legacy Information Systems: Issues and Direction by Bisbal, J.; Lawless, D.; Bing Wu; Grimson, J.; October 1999 (http://bit.ly/1sVnOYo)
[2] Software Engineering by Ian Sommerville, 10th edition 2015 (http://amzn.to/1Uo5QnB)
[3] Legacy systems definition by Techopedia(http://bit.ly/1UXRpf8)
[4] Portfolio Analysis, The business case, Aand solution for reducing risk in legacy environments by Modern Systems (http://bit.ly/22Ezw69)
[5] The burden of legacy by Dr. Toby Sucharov and Philip Rice of Erudine (http://bit.ly/1XYkSWM)
[6] Legacy systems continue to have a place in the enterprise, by computerweekly, June 2008 (http://bit.ly/XeUQ5M)
[7] How Do Professionals Perceive Legacy Systems and Software Modernization? By Utrecht University, 2014 (http://dl.acm.org/citation.cfm?id=2568318)
[8] DARWIN: On the Incremental Migration of Legacy Information Systems by Michael L. Brodie GTE laboratories and Michael Stonebreaker University of California Berkeley, March 1993
(http://db.cs.berkeley.edu/papers/S2K-93-25.pdf)
[9] Banks still handicapped by IT legacy, by computerweekly, May 2012(http://bit.ly/1U0UN6N)
[10] Moving from your legacy system- COBOL conversion, cost, and consequences bt Logapps LLC, August 2015 (http://bit.ly/24orJJ9)
[11] Who Maintains The Legacy Code by Vishwesh Bhat, October 2015 (http://bit.ly/1Y7L7du)
[12] A survey research into legacy system migration by Jesus Bisbal et al, 1997
(https://www.scss.tcd.ie/publications/tech-reports/reports.97/TCD-CS-1997-01.pdf)
[13] Security Concerns for Legacy Systems. An on-going process by Robert Annett, March 2015 (http://www.codingthearchitecture.com/2015/03/07/security_concerns_for_legacy_systems.html)
[14] Working with Legacy Systems, A Practical Guide to the Systems we Inherit and Maintain by Robert Annett. 80% complete in May 2016 (https://leanpub.com/WorkingWithLegacySystems)
{47}
References (II)[15] The Butterfly Methodology: A Gateway-free Approach for Migrating Legacy Information Systems by Bing Wu et al (http://dl.acm.org/citation.cfm?id=852129)
[16] Why legacy modernization projects fail, advice on how to how to steer your project toward Success by David Weldon, October 2014 (http://bit.ly/1t852xy)
[17] Academia needs more support to tackle the IT skills gap by Microfocus, March 2013
(http://bit.ly/22EAb7E)
[18] Microservice Architectures by Dr. Andreas Schroeder (http://bit.ly/1TOGZK8)
[19] Oracle Service Bus for Modernisation strategies by Robert Wunderlich, Ricardo Ferreira, Luis Weir, October 2015 (http://bit.ly/1ZlWtZR)
[20] Misunderstanding the Problem? By John Marshall, October 2013 (http://bit.ly/1U0TVyT)
[21] Tweet by Adrian Cockcroft (https://twitter.com/adrianco/status/542850261782237184)
[22] Microservices by Martin Fowler (minute 14), GOTO conference, Berlin November 2014
(https://www.youtube.com/watch?v=wgdBVIX9ifA)
[23] Building microservices, designing fine-grained systems by Sam Newman, October 2015
(http://shop.oreilly.com/product/0636920033158.do)
[24] Bounded context by Martin Follower, January 2014
(http://martinfowler.com/bliki/BoundedContext.html)
[25] The single responsibility principle by Robert C. Martin, November 2009
(http://bit.ly/1VDgw79)
[26] The seven deadly sins of Microservices by Daniel Bryant, January 2016
(https://www.infoq.com/presentations/7-sins-microservices)
{48}
References (III)[27] Microservice workshop – craft conference by Adrian Cockcroft, April 2015
(http://bit.ly/24os3aL)
[28] Microservice prerequisites by Martin Fowler, August 2014 (http://bit.ly/1wIjY58)
[29] Adopting Microservices at Netflix: Lessons for Team and Process Design by Tony Mauro, March 2015 (http://bit.ly/25DnkVc)
[30] The twelve factor app by Adam Wiggins (http://12factor.net/)
[31] Lehman's laws of software evolution by Meir "Manny" Lehman and László Bélády, Sep 1980 (https://en.wikipedia.org/wiki/Lehman's_laws_of_software_evolution)
[32] Conway’s law by Melvin Conway, 1967 (https://en.wikipedia.org/wiki/Conway's_law)
[33] The reactive manifesto by Jonas Bonér, Dave Farley, Roland Kuhn, and Martin Thompson, Sep 2014 (http://www.reactivemanifesto.org/)