workshop slides - introduction to dependency-oriented thinking" feb 15, 2014, sydney

(C) Ganesh Prasad, Eigner Pty Ltd

Dependency-Oriented ThinkingSession 1 – Inculcating Dependency-Oriented

Thinking


Agenda

● Introduction – Presenter bio, Participants' backgrounds

● Prior experiences with SOA● Expectations from this workshop● Case studies (interactive)● Notation (UML) for inter-system dependencies● Surrogate principles and why they don't always

apply● Exercises (interactive)


SOA is not just Technology● Technology is the implementation of business intent

● A lot of analysis and design is required before business intent can be implemented


How did SOA get to be seen as Technology?● “Service” is a weasel word


Understanding the term “dependency”

● Not a fancy technical term; just a regular English word

● We can represent it using a formal notation


Why are dependencies important?

● A system with a large number of interconnections (dependencies)

● Some of the dependencies are not even obvious

● Now move the orange block without disturbing any of the others

● How confident are you?


The impact of reducing dependencies● A system with far fewer interconnections (dependencies)

● No dependencies are unknown or hidden

● How confident are you of making changes without breaking anything?

● How fast can you move the block?

● How much do you need to plan for the move?


The trap of surrogate principles

● “Point-to-point connections considered harmful” considered harmful


The converse trap

● Mediated connections considered harmful


Logical mediation

● A “point-to-point” connection can in fact be mediated


Dependency view● The dependency view of a true point-to-point connection


Dependency view● The dependency view of a mediated “point-to-point” connection


Logical tight coupling● A “mediated” connection may in fact be tightly coupled


Dependency view

● The dependency view of a tightly coupled mediated system


Reducing dependencies – the Linux way

● Linux uses the Unix mechanism of symbolic links to abstract changes


Dependency-Oriented ThinkingSession 2 – Dependency-Oriented Thinking as a

Formal Approach


Agenda

● The BAIT Model of the enterprise● Entities and Relationships in the TOGAF Model● The core elements applicable to Analysis and

Design (Design Artifacts)● Dependency Principles● The Method


A Few Words on BAIT


How should an architect view “applications”?● The literal view of systems does not provide architectural insights

● The Application layer of BAIT is to be viewed functionally, not physically


The Layered view of Applications● The correct way to see the Application layer is as logical groupings of

business capabilities

● Physical systems (hardware and software) are Technology layer entities used to implement these capabilities


Interdependencies between BAIT layers● The Application and Information layers always go hand in hand


BAIT as a “Four I” model of SOA

● Intent – what is to be done

● Internal Cohesion – how functions hang together

● Interfaces and Integration – how functions interoperate

● Implementation – what the tangible, working system looks like


BAIT and “Operation-Oriented Architecture”

● Operations are the fundamental building blocks of an enterprise

● Determine operations (the steps of business processes)

● Group related operations together (based on internals and interfaces)

● Expose operations to external consumers

● Execute operations, either alone or in combination with others


BAIT versus TOGAF

● BAIT is only a framework● It shows how to view an enterprise but doesn't

show what comprises each layer● We need a model that is based on BAIT to

define a generic set of entities at each layer● A good model to use is TOGAF● TOGAF is based on industry experience and

practice, and is comprehensive yet generic


TOGAF as a well-known set of dependencies● TOGAF's vocabulary includes “viewpoints”, which are subsets of the

entities that make up the model of an enterprise

● We can mine this rich set of data to search for dependency relationships


Enterprise-Level Entities and Relationships


Business Layer Entities and Relationships


Business Layer Design Artifacts

● Not all Business Layer entities are important to an analyst or designer


Application Layer Entities and Relationships


A Detailed Look at “Applications”


Application Layer Design Artifacts

● Note the two types of “applications” formed by grouping Operations in two different ways


Information Layer Entities and Relationships


Information Layer Design Artifacts


Technology Layer Entities and Relationships


Technology Layer Design Artifacts


Design Artifacts by BAIT Layer


Dependency Principles by BAIT Layer


The Formal Dependency-Oriented Method● Starting at the Business layer, identify the Key Design Artifacts and

apply Dependency Principles to arrive at an optimal design

● Rinse and repeat at the next lower level


Dependency-Oriented ThinkingSession 3 – The Business Layer


Agenda

● Business Intent● The Chain from Vision to Process Step● Traceability and Minimalism● The role of Domain Insight● Dealing with changing requirements (intent

versus detail)


Business Layer Design Artifacts


Vision and Mission

● Vision – your idea of Utopia● Mission – your role in bringing about that Utopia● Together referred to as “Goal” in TOGAF


The hidden role of Strategy● Strategy shapes organisations

● The same Mission may be accomplished in different ways


Business Functions are largely Stable

● Changes in Strategy tend to impact Business Processes, but curiously, need not impact Business Functions once established

● Changes in Business Function are known as “restructures”


Function, Process and Process Step

● Function is both Structure and mini-Mission● Processes are what Functions “do”● Processes are composed of Steps (which may

or may not be reusable)


The Traceability Principle● Think of Traceability as a chain of “existential dependencies”

● It's a discipline that makes a business “lean and mean”


The Minimalism Principle● “The simplest way to get a job done”

● A discipline that leads to agility


Domain Insight

● Helps in applying the Traceability and Minimalism principles

● More Art than Science● Think of examples from your own organisation


Changing Requirements

● Intent versus Detail


Dependency-Oriented ThinkingSession 4 – Process Design


Agenda

● The generic business process● Human workflow and asynchronous process

steps● Orchestration and Choreography● "Orchestrable" and "Choreographable"

Operations


The Crucial Design Step between Process and Process Step

● How should Process Steps be coordinated into a Process?

● Design Exercise


Top-Down Design of the Business Layer


Orchestrated Processes● The dependency relationships in an orchestrated process mirror the

flow of control


Choreographed Processes● The dependency relationships bear no resemblance to the flow of

control


A Simple Business Process

● An Event Trigger followed by one or more Process Steps


A Generic Business Process

● A generic process does not run to completion in one go

● It may pause, and resume based on intermediate event triggers


“Non-blocking” Processes and Callback

● It's often efficient for a paused process to do something else instead of idly waiting for an event trigger

● This model enables the design of long-running processes


Human workflow as a long-running process● Non-blocking process design can be used to design human workflow

● Pause and Resume junctures occur wherever human interaction is required


Dependencies in Orchestrable and Choreographable Operations

● Recall the different graphs for dependency versus flow of control

● The dependencies are similar, just in different places


The Overhyped Notion of “Reuse”

● Reuse is not a first-class benefit● It's unreasonable to expect reuse from a SOA

initiative if the process steps of a business are not inherently reusable


The potential for reuse is first seen at the Business layer


Lower layers should confirm if reuse is in fact possible


Dependency-Oriented ThinkingSession 5 – The Application Layer


Agenda

● What is an "Application"?● The principles of High Cohesion and

Decoupling of Internals from Interfaces● "Goldilocks" Signatures (Stability versus

Precision of Interfaces)● Variants, Versions and dealing with change● Shared Semantics for effective Choreography


Application Layer Design Artifacts● Process Steps at the Business layer map to Operations at the

Application layer


Application Layer vs Technology Layer● The Technology layer provides a detailed “street view”

● The Application layer provides a high-level “map view”


The Relationship between Design Artifacts at the Application and Information Layers

● Products are defined by a shared Internal Data Model

● Services are defined by a shared Interface Data Model


The Cohesion and Coupling Principles● At each layer, there is an internal focus (Cohesion) and an external

focus (Coupling)


Cohesion Exercise 1


Cohesion Exercise 2


Cohesion Exercise 3


Simultaneous Groupings● It is possible to group entities in more than one way at the same time


Simultaneous Grouping of Operations● A dynamic grouping of Operations is a Process

● A static grouping of Operations is an Application (Product or Service)


Anatomy of a Product● Grouping by internals is the design principle

● External access is “canned” through a relatively rigid native interface


Anatomy of a Service● Grouping by Interfaces is the design principle

● External access is the very raison d'être!


SOFEA – Architecture for Front-ends● For the best of both worlds, build user interfaces on top of services


Demystifying Versioning

● Variants and Versions

● Interfaces and Internals

●


Variants● A Variant is an operation “flavour” that may coexist with other flavours

without deprecating them


Versions

● There is always a latest Version that deprecates earlier ones


Interfaces as Abstractions of Operations● An Interface represents the Business Intent of an Operation

● Details are taken care of by implementation variants (“internals”)


The Impact of Changing Interfaces

● Consumers of operations have a dependency on the interface

● Changes to operations may impact consumers to different degrees


Impacts classified by Version Type● Versions can be thought of as “x.y.z”

● The only criterion for classification is the level of impact from a change

● A term like “bug fix version” makes no sense from a dependency perspective


Changes to Internals● A change to the internal logic of an operation with no visibility at the

interface changes only the implementation version, not the minor or major versions


“Minor” Changes to Interfaces● Some kinds of changes to operations are visible at the interface, but

do not impact consumers in a practical sense

● They only change the minor version number

● Minor version changes are “backward compatible” (continue to support older consumers while supporting new ones)


Types of changes that do not break an interface

● Interfaces that become more lenient tend not to impact consumers


Major Changes to Interfaces● A change to an operation that is not only visible at the interface but

also forces a change to the consumer is a change to the major version number


Types of changes that break an interface● An interface that becomes stricter or more restrictive tends to impact

consumers


Version Mapping Convention● Exploit the dependency level expressed by the “x.y.z” version numbering

scheme

● The minor version number should always refer to the latest implementation version


Version Mapping Convention, cont'd● The major version number should always map to the latest minor

version


Version Routing Exercise● Which implementation version will be executed by this call?


Solution● Thumb rule – It's always the latest version at the lower level


The “Minor Version Lockstep Dependency”● When dealing with multiple variants, changes to the minor version

number of one variant will require corresponding changes to the minor version numbers of all the other variants!


The “Goldilocks Signature” Principle● A balance is required between the stability and the precision of an

operation's interface


What is a “Well-Designed” Interface?

● Well-designed interfaces are stable in the face of change

● i.e., they do not create unnecessary dependencies


Why the scheme works


Managing Breaking Changes


Managing Breaking Changes, cont'd


Managing Non-Breaking Changes


Dependency-Oriented ThinkingSession 6 – The Information Layer


Agenda

● "Data on the Outside versus Data on the Inside"

● Elements of Low Coupling● Type Hierarchy and Interface Abstraction● Identity Association● Context versus Content


Information Layer Design Artifacts

● Only two major categories – “Data on the Outside” and “Data on the Inside”

● Deceptively simple!


Variants from a Data Perspective● A single abstract interface can map to multiple concrete

implementation variants

● But the -------- ------ remains the same


The Notion of “Context”

● “Context” is a way of formalising aspects of an operation that are not core to its business intent


Low Coupling, or “Need to Know”

● A form of minimalism applied to data


Decoupling of Internal Data from Interface Data

● To be decoupled, interface and internal data models cannot have dependencies on each other


The Domain-Specific Data Dictionary

● Efficiencies are still possible!

● The internal and interface data models do talk about similar entities


What a Data Dictionary Looks Like

● Entities and Value Objects are defined here


Types and Instances● Both internal and interface data models define instances of types

● Types common to both should be in the common data dictionary


Data Model Example● Internal and interface data models have some elements in common

and some elements that are unique to each


The Relationship between Interface/Internals and Type Hierarchy

● This is why Application and Information layer design is done together!


A Problem We Glossed Over!

● Abstract interfaces are stable and shield consumers from many internal changes, but...

● How does a consumer know which concrete data elements to send and receive?

● The answer becomes clear if we jettison a fallacy of SOA – that a formal “interface contract” is sufficient for a consumer to begin transacting with an operation

● In practice, designers of consuming applications always discuss with the designers of operations and negotiate data exchange

● Use this knowledge to design pragmatic design processes


Building an abstract interface to an existing implementation


When a new consumer is supported by an existing variant


When a new consumer is not supported by an existing variant


The Internal Data Model● Deals with Entities rather than Value objects


The Interface Data Model● Never references Entities! (That would be tight coupling)

● Value objects are packaged into messages that are exchanged

● Value objects can also be called “representations”


The Identity Association Principle

● A common design error


How Identity Association Works

● Consumers must not develop dependencies on internal data entities


Revisiting the Goldilocks Signature Principle● Think about stability and precision afresh from the viewpoint of data


Message Data Model

● The Goldilocks Signature principle is supported by this model


A Terrible Way to Implement Extensible Interfaces

● Extensions gradually destroy structure and meaning


Extensibility with Control● Setting up a Type Hierarchy supports change gracefully


Isolation of Context from Content● Keep extraneous considerations separate from the core business

intent


The Context Type Hierarchy● Context has its own hierarchy with a single, optional base type called

“Context Type”


Supporting New Variants that Differ only in Context

● Building in an optional context element into every interface is insurance against breaking change


Context of Output Data● Context of input data is used to determine appropriate variants to

invoke

● Context of output data can support choreographable operations


Service Content and Process Context● When considering a single operation, some data elements may be

considered content rather than context (i.e., they follow from the business intent)


Modelling Service Content as Context

● Sometimes, a process may have operations with common content

● This content could be considered to be their common context


Context By Reference● Operations sharing “content context” with other operations can then deal

with references to common data

● These reference IDs are a form of context that is common to the process


Context By Reference, cont'd

● Subsequent operations refer to common content through the context ID


Example – Bringing Application and Data Layer Principles Together


The Type Hierarchy supporting Late Binding


“Schema Validation”


Routing/Binding from Interface to Implementation


Dependency-Oriented ThinkingSession 7 – The Technology Layer


Agenda

● Standards, Bundling and Tooling● Standards and Standards Families - SOAP and

REST● Bundling - Data, Logic, Physical Components

and their association● Tooling - the core and supporting components

of a distributed solution● Implementing a logical design


Technology Layer – The Design of Implementation

● Not implementation itself, but the design of implementation


Technology Layer Design Artifacts● At the most abstract level, Technology consists of three elements

● They are Standards, Tooling and Bundling


Bundling Decisions

● “Where to place what” is a bundling decision


Logic Bundling Principle● This is a variant of the Cohesion principle – what belongs together

must go together


The State or “Stickiness” Principle● Placing logic on one physical component as opposed to another could

be the difference between a stateless and a stateful system

● Stateless systems have significant architectural advantages on account of their reduced bundling dependencies (scalability, failure recovery)


Topology “Hotspots”● The way physical components are connected or “wired” together can

create bundling dependencies of another kind

● Performance bottlenecks and single points of failure are common results of topology-related bundling decisions


Reference Architecture of a Deployable Application


Determining Standards for Orchestrable Operations


Determining Standards for Choreographable Operations


Determining Tooling Components


Determining Deployment Bundles


Determining Description Bundles


The Service Provider


The Service Consumer


Legacy Systems


Functional Symbols for Tooling● From Gregor Hohpe's “Enterprise Integration Patterns”


Additional Symbol for Tooling● Internal logic has no symbol in “Enterprise Integration Patterns”

● We have to “roll our own”!


The Core Technology Layer Components

● Service Container

● Broker

● Process Coordinator


Service Container● Implements Business Logic and exposes it


How a Service Container is Used

● The logic implemented by the Service Container and exposed as an operation is consumed by a service consumer


Broker

● Mediates access to existing logic


How a Broker is Used● The special protocols and interfaces of legacy systems are hidden

through “adapters”

● Data formats are converted through “transformer” logic

● Physical locations are hidden and variants are abstracted through “mediator” logic


Process Coordinator

● Combines existing operations into business processes

● More relevant for orchestrable operations


How a Process Coordinator is Used● The Process Coordinator runs stateful process logic (orchestration)

● It invokes operations as required

● The operations themselves are stateless


The Components Working Together

● The three core technology components are designed to work together


Broker Design Error 1● Topology Hotspot – the Broker is not meant to be a centralised

component


Broker Design Error 2

● Always use the right tool for the job

● The Broker is a mediator

● Do not use the Broker as a Service Container

● Do not use the Broker as a Process Coordinator


Dependencies introduced by SOAP● Static Description dependency

● Bundling dependency

● Synchronisation dependency


Dependencies introduced by REST● Semantic dependency

● Synchronisation dependency


What is a “Semantic Dependency”?● The requirement for a system to resolve ambiguity


The Problem with WSDL

● WSDL is a static description of a group of operations

● Three dependencies arise from this design


The WSDL Dependency Multiplied● A popular service can have many consumers that are then dependent

on its interface contract


Cascading Impacts of Change● WSDL is a dependency hotspot


Calculating the Brittleness of WSDL● A WSDL file describes exactly 1 service consisting of 5 operations, each

with an input and an output document

● If any given document has a 10% probability of changing, what is the probability of the WSDL file changing?


Strategies to minimise change● Dependency principles can be applied to minimise the effects of

change due to the WSDL dependency hotspot


Dependencies arising from REST's Hypermedia Constraints

● There is no static description of operations in REST (WADL is not RESTian)

● There are still 3 dependencies


Miscellaneous Technology Layer Considerations


The “Canonical Data Model” Fallacy● A Canonical Data Model could be too heavyweight and rigid


Domain Data Models● Domain Data Models are more flexible and federation-friendly


“The SOA Ecosystem”● The logical view of service providers and consumers


Processes as Service Consumers● Processes are a special type of service consumer


Processes as Operations

● A process abstracts coarse-grained logic

● This coarse-grained logic may in turn be invoked by a service consumer


Processes in the SOA Ecosystem

● A process is both a service provider and a service consumer


Operation Signatures implemented in SOAP and REST


Dependency-Oriented ThinkingSession 8 – Worked-Out Examples


Agenda

● Option 1: Detailed analysis of participants' own design problems

● Option 2: Canned case studies:● Conducting a post-mortem based on dependency

principles● Designing a system based on an orchestrated business

process● Designing a system based on a choreographed

business process● Designing an interface data model to cater to change

(Type Hierarchy and Goldilocks Signatures)


Post-Mortem Example


Exercise – Post-Mortem

● What is wrong with this design?


A More Flexible Design


Synchronous Orchestrated Process


Banking Application Sequence Diagram


Processes, Services and Operations

● Dynamic grouping of operations into a single process “Open Account”

● Static grouping of operations into 3 services


High-Level Solution Diagram – Banking


Asynchronous Orchestrated Process


Insurance Application Sequence Diagram


Processes, Services and Operations● Two dynamic groupings (processes): Get Quote and Purchase Policy

● 6 static groupings (services)


High-Level Solution Diagram – Insurance


Choreographed Process


A Maze Game


Allowed Movements


Cell Identifiers – Identity Association● A cell's location must not be derivable from its name or vice-versa


The Design – Hypermedia Constraints


Interaction at the start of the game


Interaction in the middle of the game


Interaction at the end of the game


Managing Change


The 70s● Banking in a simpler (and more rigid) time


Service Interfaces to Support Arbitrary Clients


Variants of Similar Operations


Standardised Variants, Abstract Interfaces


Processes as Standardised Operation Variants


Supporting a New Variant


Deployment of Components


A More Likely Scenario!