the analysis part of integration projects
TRANSCRIPT
![Page 1: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/1.jpg)
Connected Systems Consulting Ltd
Sponsored & Brought to you by
The Analysis Part of Integration ProjectsMichael Stephenson
https://twitter.com/michael_stephen
https://www.linkedin.com/in/michaelstephensonuk1
![Page 2: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/2.jpg)
The Analysis Part of Integration Projects
Connected Systems Consulting Ltd
![Page 3: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/3.jpg)
Connected Systems Consulting Ltd
Who am I?Michael Stephenson
– UK-Based Freelance Consultant specializing in:• BizTalk• Windows Azure• Integration
– Microsoft Integration MVP for 6+ years– Pluralsight Author– Azure Insider/Advisor– One of organizers of UK Connected Systems User Group– Founder of BizTalk Maturity Assessment – www.biztalkmaturity.com– Worked on approx. 30 projects that have leveraged Azure– First project went live about 4 years ago
– Contact Info:• Blog: http://microsoftintegration.guru• Twitter: @michael_Stephen• Linked In: http://www.linkedin.com/in/michaelstephensonuk1• Email: [email protected]
![Page 4: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/4.jpg)
Thanks to….
![Page 5: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/5.jpg)
Question
“How often do you feel thrown in at the deep end when starting an integrationdevelopment project because no one seems to have a clear understanding ofwhat is required”?
+1 in the chat window if you think this is common
![Page 6: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/6.jpg)
The Usual Story
![Page 7: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/7.jpg)
Common Scenario 1
“I need a feed of customer data from system A to system B”
Err…. Ok so what does this mean?
![Page 8: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/8.jpg)
Common Scenario 2
Contoso
Fabrikam
Acme
CRM System
ERPSystem
FinanceSystem
MarketingSystem
ProductsSystem
ProductsSystem
OrdersSystem
OrdersSystem
OrdersSystem
CustomersSystem
Its this interface here, can we just do this?
Those arrows are pretty but we don’t know anything about what it means
![Page 9: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/9.jpg)
Common Scenario 3
“I need an additional data attribute added to an Existing integration process”
This may be straightforward and we could just get on
with it… but lets get a little info to be sure
![Page 10: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/10.jpg)
Common Scenario 4
“We need to change the existing new customer processSo it also updates the new CRM system”
I hope we have all of the data we
might need
![Page 11: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/11.jpg)
The Problem
![Page 12: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/12.jpg)
What systems are involved?
What does the data look like?
How much data do we
need to process?
What do things happen?
What is the process flow?
Are there any alternative or
exception scenarios?
How do I talk to each
application?
Are there any SLA’s?
Is there any business logic?
What are the data
transformation rules?
Will we need any new
infrastructure?
Are there any performance
requirements?
Are there any security
requirements?
Is there any documentation
?
When do things
happen?
![Page 13: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/13.jpg)
“There’s just so much unknown that affects how we design and build the system”
![Page 14: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/14.jpg)
How Important is Analysis?
Integration is easy when…• The solution is very like something you have done before
• You are reusing stuff you have build in the past
• The applications involved are well understood
• Only a small number of people are involved
• The data is not complicated
• The process logic is simple
• You have a good test environment setup
• Everyone understands the process to be built
• You have good testing
• Your dependencies are well understood
Integration is hard when…• You need new infrastructure
• You have lots of dependencies or they are poorly understood
• Lots of external vendors or people are involved
• You need to use patterns you have not implemented before
• The applications involved are not well understood
• The applications involved are unreliable
• You have a poor test environment setup
• You follow good ALM practices
• You don’t have enough information
• Your requirements are not well understood or clear
• You have a poor or incorrect solution design
• Your making it up as you go along
![Page 15: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/15.jpg)
Getting it wrong
Not enough analysis• Developer gets things wrong
• Architect makes bad assumptions
• Poor documentation
• Testing problems
• Delays to redo and fix things
• Developer ends up chasing around to do analysis that wasn’t done
• Takes longer to deliver value
• Unhappy team
• Unhappy customer
Too much analysis• Wasted time
• Too much documentation
• Lots of the information doesn’t add value
• Takes longer to deliver
• Costs too much
• Over complicates things
• Difficult to deal with change
• Time to delivering value is high
• Unhappy customer
![Page 16: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/16.jpg)
Amount of Analysis
Amount of Analysis
High
HighLowSweet spot = Just enough
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 300
20
40
60
80
100
120
Chart Title
Cost of Project Risk
We are usually in one of these places
![Page 17: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/17.jpg)
MethodologiesI have a methodology doesn’t this solve the problem?
![Page 18: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/18.jpg)
Methodologies – In Theory
Waterfall Agile
Analysis
DesignDevelop
Analysis Design Develop
![Page 19: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/19.jpg)
Methodologies – In Practice
Waterfall AgileAnalysis & Developme
nt
Re-doing Analysis & Develop that
we missed or got wrong
Analysis Design Develop
Analysis Design Develop
Time Spent
Value Delivered
What happened to architecture?
![Page 20: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/20.jpg)
Agile v’s Waterfall
Agile• Just give it to the scrum team
• We are doing agile so things are assumed to be simpler
• The team will figure it out
• Often teams don’t do analysis sprints
• Often This should be a sprint 0 type activity
• You usually have more stakeholders than just the product owner
• Encourages POC of risk areas which is good
Waterfall• Lots of analysis and design before any real world
validation
• Analyst/Architect not close enough to delivery to handle change
• Analysis sometimes too detailed and overkill in terms of presentation format
• Analysis that doesn’t add value
“In the real world a methodology often doesn’t solve the problemas they aren’t getting the right amount of the right information”
![Page 21: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/21.jpg)
Doing it effectively
Lets take the best things and make
them work?
![Page 22: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/22.jpg)
“Gathering the information we need to allow us to design, build and test the solution the customer needs, and being able to communicate that information to the entire team”
Analysis is….
![Page 23: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/23.jpg)
“Just enough analysis to do an effective job”
“Analyst and Architect working together”
![Page 24: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/24.jpg)
The Project
Developer
Legacy ApplicationTeam
ERP System Team
Infrastructure Team Support Team Security Team
Finance Team
Project ManagerSenior Management
Change Team
![Page 25: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/25.jpg)
Definition of Ready
Do we feel ready?• Do we feel confident about what
we need to do• Is the project broken down into
stories or features?• Do we need or have the right
analysis artefacts
Measuring Readiness• INVEST• I – Independent (of others)• N – Negotiable• V – Valuable• E – Estimatable• S – Small• T - Testable
![Page 26: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/26.jpg)
Analysis Iterations
• Iteration 1 – High Level• Iteration 2 - More Detail• Iteration 3 – Reduce Risk
Iteration 1
Iteration 2
Iteration n
![Page 27: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/27.jpg)
Iteration 1 - Analysis
![Page 28: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/28.jpg)
Step 1 – The User Story
StructureIn order to (why is it important)As a ………. (who wants it)I want …… (what do you need)
ExampleIn order to collect money from customers
As a sales director
I want the CRM system to process credit card payments on a monthly basis
“If you cant explain the value statement then why do we want to do it”?
![Page 29: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/29.jpg)
Step 2 – The Context Diagram
Old CRM New CRM
Orders Website
Lookup Customer
Customer Transfer
Orders System
Process Orders
![Page 30: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/30.jpg)
Step 3 – Integration Use Cases
![Page 31: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/31.jpg)
Step 4 – High Level Process Logic
• Could you easily explain the process logic to someone
![Page 32: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/32.jpg)
Step 5 – High Level Data Items
• What entities?
• Any relationships?
• Any key data attributes?
• Where does data come from?
![Page 33: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/33.jpg)
Step 6 – Applications Involved
• What are the supported interfaces?
• What protocols are supported?
• Where are the applications?
• Who manages them?
• Who are the key contacts?
![Page 34: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/34.jpg)
Step 7 – Non-Functional Requirements
• Is there a general list of NFR’s
• Are there any specific NFR’s raised by the business
• Are their any IT NFR’s
• How should the system perform?
![Page 35: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/35.jpg)
Step 8 – Stakeholder Map
Keep Satisfied
• CRM Development Team
• Infrastructure Team
Actively Engage
• Product Owner• HR Department• Support Team
Monitor
• Warehouse Dept
Keep Informed
• ERP Development Team
• Finance Dept
• Make sure we identify stakeholders and where they fit into the map so we can make sure we talk to the right people at the right time
• Make sure owners of error conditions are identified not just owners of processes
![Page 36: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/36.jpg)
Step 9 – Which type of Integration?
• API & Services
• ESB / Messaging
• Batch Based
• Simple Orchestration
• Complex Orchestration
• Mixed
![Page 37: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/37.jpg)
Step 10 – Candidate Architecture
• Is it something we already do?
• What patterns do we need?
• How might we build it?
• How confident are we about the design?
![Page 38: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/38.jpg)
Step 11 – Create Integration Catalogue• Agree a friendly Interface Name
• Identify Source and Destination Systems
• Identify Candidate Integration Type• Batch• API• ESB/Messaging• Complex Orchestration
• Identify Types of Data involved
• Identify grouping of interfaces
• Identify related Use Cases
• Relate non functional requirements
• Relate candidate architecture
![Page 39: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/39.jpg)
Checklist
What have we done• Created an Integration Catalogue for project which
pulls together:• User Stories• High Level Context• Identified Use Cases• Draw Process Logic• High Level Data Model• Applications Involved• Stakeholders• Non-Functional Req’s• Candidate architecture
How do we feel• Are we confident about process
• Are we confident about data
• Are we confident about candidate architecture
• Does it feel simple or complex
• Do we feel we have a good understanding
![Page 40: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/40.jpg)
Iteration Summary
![Page 41: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/41.jpg)
Iteration 2 – Analysis & Design
![Page 42: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/42.jpg)
“We have decided we need more information & detail before proceeding, what do we get next?”
What next?
![Page 43: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/43.jpg)
Where do we look – Key Areas
1 Processing Logic • What processing logic is required
• What data transformation is required
• Are there any patterns
2 Application Connectivity
• How might we connect to the application
• Is the application interface well defined and easy to work with
3 The Data • Does the data make sense• Does the data map between
systems well
1
2
2
2
2
3
3
3
![Page 44: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/44.jpg)
Step 1 – Flush out features
• Find alternative scenarios
• Specification by example
• Find exceptions
• Confirm owner of exception scenarios
![Page 45: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/45.jpg)
Step 2 – Define Data
• Get sample data
• Data Model
• Get data specifications if existing• XSD/XML• WSDL/Swagger• JSON• Flat File Structure• HL7• EDI
![Page 46: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/46.jpg)
Step 3 – Define Data Mappings
• Encoding formats
• Reference data mappings (Look up values)• Mr AA• Mrs AB• Miss AC
• Number formatting• Leading zeros• Decimal places
• Text• Length• Alignment• Padding
• Date formatting
Important this can often be an areaWhich can throw up more integration use cases or maintenance challenges
![Page 47: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/47.jpg)
Step 4 – Application Integration Capabilities
• Do we have connectors?
• Are the connectors well understood?
• Are there challenges doing the connection?
• How will security work?
• Are there any throttling requirements?
![Page 48: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/48.jpg)
Step 5 – Non Functional’s
• Revisit the NFR’s and validate with stakeholders
![Page 49: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/49.jpg)
Step 6 – Patterns
• Are there any obvious integration patterns involved
• Patterns can help create a common understanding
![Page 50: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/50.jpg)
Step 7 – Design Decisions
• Are there any key design decisions to make• Who will be involved• How will the decision be made• How do we communicate to the decision to everyone• How do we ensure the decision is followed?
![Page 51: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/51.jpg)
Step 8 – Update Candidate Architecture
• Is our candidate still good?
• Are there any new things
![Page 52: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/52.jpg)
Checklist
What have we done• More detailed process definition using
Specification by Example
• Defined data in more detail
• Define mappings
• Reviewed NFR’s
• Identify Patterns
• Update Architecture
How do we feel• Are we confident about process
• Are we confident about data
• Are we confident about candidate architecture
• Do we understand the complexity
![Page 53: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/53.jpg)
Iteration Summary
![Page 54: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/54.jpg)
Iteration 3 – Reduce Risk
![Page 55: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/55.jpg)
“How do we reduce the risk around the areas we have concerns with”
What next?
![Page 56: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/56.jpg)
Step 1 – Logical Process POC
• What to do• Implement light weight stubs to prove the applications will
work
• What does it achieve• Allows focus on “will the applications work”• Mitigates risk that the process logic will not work
![Page 57: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/57.jpg)
Step 2 – Application Integration POC
• What to do• Test we can connect to the application• Create BDD style Specflow tests of the Application interface
• What does this achieve• Test the application behaves the way we expect it to• Tests the data looks like what we expect it to• Tests assumptions about error conditions
![Page 58: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/58.jpg)
Step 3 – Update Candidate Architecture
• Is our candidate still good?
• Are there any new things
![Page 59: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/59.jpg)
Checklist
What have we done• POC the key risk areas
• Updated architecture candidate
How do we feel• Do we have any concerns?
• Have we mitigated risks?
![Page 60: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/60.jpg)
Iteration Summary
![Page 61: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/61.jpg)
Analysis Iterations
• Just enough analysis to set us up to be successful
• Use the iterations that are needed
• Make sure information is clear and can be understood by anyone in the team
Iteration 1
Iteration 2
Iteration n
![Page 62: The Analysis Part of Integration Projects](https://reader030.vdocuments.net/reader030/viewer/2022032618/55b6e6bebb61eb6e688b46d2/html5/thumbnails/62.jpg)
Takeaway
• Define your organisations definition of ready?• What analysis artefacts do you usually need?• Share it?• Who is up for some community guidance around this?• Technet wiki or something?