data modeling in the api economy

Data Modeling in theAPI EconomyPRESENTATION TO DATA MODELING ZONE, OCTOBER 2014

TED EPSTEIN, FOUNDER AND CEO

MODELSOLV, INC.

COPYRIGHT © 2014, MODELSOLV, INC. | ALL RIGHTS RESERVED. 1

All Eyes on APIsParadigm shift with integration architecture, happening on two fronts:◦ REST as a recommended architectural style for internal services

◦ Emergence of the API Economy


Internal RESTWeb-Style REST APIs are replacing SOAP for internal services.

REST brings some useful design constructs for data-oriented APIs.

REST is another wave of change in integration architecture◦ New trends bring new projects, modernization programs

◦ A chance to improve on whatever we did before.

◦ In this case, a chance to improve on SOA

We need you!◦ Large scale integration needs data modelers!

◦ We need to make the case for that, and get our IT organizations on a better path.

◦ This talk will help get you started.


Emergence of the API EconomyAPIs have entered the mainstream of business and marketing awareness.

Business can profit from APIs◦ As products and services.

◦ As sales channels.

◦ As points of customer engagement.


Massive Integration on Web and MobileEverything integrating with everything else◦ Mashups

◦ Common Services◦ Identity (Google, Facebook, Twitter, OpenID)

◦ Transactions (PayPal, Google Wallet, Amazon Payments)

◦ Location (Google Maps)

◦ Document Sharing (Dropbox, Google Drive, SkyDrive)

◦ Social (LinkedIn, Facebook, Twitter)


Integration Trends in Business and GovernmentCloud/SaaS

Big Data

Mobility

Internet of Things

Open Government


Mapping the Service Landscape


Internal and External ServicesInternal: We're somewhere on the REST adoption curve.◦ REST is the new SOA

◦ No, it's an evolution of SOA◦ No, it's different, for this reason or that reason◦ Just don't call it SOA, because we don't like SOA◦ Fine, but it's still the new SOA◦ ... and that's OK. Really.

External: We're jumping on the API bandwagon.◦ May be headed by marketing◦ May be a separate technology team


Meeting in the Middle


source: http://www.soatothecloud.com/

There is no hard line:◦ We know that some

private APIs will become public.

◦ We don’t know which ones, or when.

◦ We should be converging towards a common architecture for internal & external services.

What's gov got to do with it?Why do service APIs need “governance”?

APIs are data conduits: ◦ Services account for a significant portion of data flow

Major technology initiatives are integration-intensive:◦ Mobile, Big Data, Cloud Computing /SaaS

◦ Regulatory compliance

Integration is the #2 challenge in Mobile and SaaS:◦ #1 is data security, #2 is integration


sources: MuleSoft and SnapLogic surveys

Why is integration such a headache?Our systems are not speaking the same language.

Every time a new team builds a new API, the IT landscape gets more complex.

Billions spent on remedial solutions to aggregate disparate data, but we have a leaky roof.◦ Increasing complexity◦ Poor data quality◦ High integration cost◦ Slow time to market


What are we doing to stop these leaks?This is why data modelers need to get involved.

We need unified canonical data models in our APIs.

We need data quality constraints.

Governance and Collaboration: two complementary paths to convergence.


REST Standards -Exploring the Stack


A Simple Data Modelstructure TaxFiling

id : stringyear : gYearnetTax : decimalcurrency : stringtaxpayer : reference to Person

structure Person taxpayerID : stringlastName : stringfirstName : stringaddresses : reference to Address*

structure Address street1 : stringstreet2 : stringcity : stringstateOrProvince : stringpostalCode : string


REST: An Architectural StyleSome Essential Modeling Constructs:

◦ Resources

◦ Identifiers

◦ Representations

◦ Media Types

◦ Methods

◦ Hyperlinks


Fielding DisclaimerStrict REST vs. REST in Practice◦ URIs: opaque vs. transparent

◦ Out-of-Band Information: media types, link relations vs. documentation

◦ Processing Model: hypertext-driven vs. client-driven

http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven


http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven

Data Model as Resource Model


CollectionResource

Data Type

ObjectResource

HTTPSome Essential HTTP Modeling Elements:◦ URIs

◦ Methods

◦ Headers

◦ Status Codes


REST/HTTP Resource Model


URI Template

Method

Request

Response

Status Code

Media TypesAbout Media Types◦ MediaType(Resource) --> Representation

◦ Appears in the content-type HTTP header

◦ Used for content negotiation

◦ Standard media types registered at:

http://www.iana.org/assignments/media-types/media-types.xhtml


http://www.iana.org/assignments/media-types/media-types.xhtml

Low-Level Media TypesEssentially Wire Formats◦ Technology-specific

◦ Domain-neutral

Examples:◦ application/xml

◦ application/json

many others◦ text/html

◦ image/jpeg

◦ application/protobuf

◦ …


Generic Structured Media TypesWhat are they?◦ Data-oriented media types a level above basic wire formats

◦ Many of these basic formats are not hypermedia-aware

◦ Intended to express generic structured data, but with some additional abstractions:Hyperlinks, embedded representations, collections, items…

◦ Different media types will emphasize different things


HAL


Sidebar: Hyperlinks◦ Hyperlinks make REST genuinely different.

◦ Hyperlinks are your friend.◦ A better way to model relationships◦ Tends to de-emphasize big, deeply nested messages◦ Helps neutralize some of the perspective mismatch problem◦ Hides identifiers in hyperlinks

◦ Isn't that expensive?◦ Cache can go a long way

◦ When to embed vs. hyperlink:◦ For network efficiency (even with caching)◦ For client developer convenience, if the embedded data will most often be required◦ For encapsulation (Think DDD Composites)


Hyperlinked Resource Model


Hyperlink

Domain-Specific MetadataOptions◦ Domain-Specific Media Types (Sun Cloud, Atom)

◦ Schemas (XSD, JSON, etc.)

◦ Profiles

◦ Microformats

◦ Documentation


Diagram with Domain-Specific Data


Sidebar: Schemas and RESTA cultural issue: uncomfortable relationship between REST and formal domain-specific data contracts.

XML Schema was a sore spot for SOA.

Domain-specific data contracts present a dilemma for strict REST:

◦ Out-of-band information is supposed to be concentrated in media types and link relations,

◦ ... but we don't want an explosion of media types.

REST is silent on the question of what to do with domain-specific data.

REST is silent on whether your payload can or should include a schema.

An opinionated perspective:

◦ There's always a schema, whether you document it or not.

◦ There are new options (JSON Schema, technology-independent schemas).

◦ Defining a data contract formally just makes sense, much of the time.


API Description LanguagesOptions◦ Swagger

◦ API Blueprint

◦ RAML

What they have in common◦ Allow modeling of resources, data, methods, links, etc.

◦ Public domain specifications, open-source web-based tooling.

◦ Generate interactive documentation / testing sandbox.

◦ Integration libraries and code generators.

◦ Highly recommended as learning tools, to help you explore how REST APIs are put together.


Beyond REST as a Wrapper


REST as a Silver BulletThe fantasy:◦ Exactly one resource for everything

◦ The golden source is the only source

◦ Everything perfectly hyperlinked

The Reality◦ One size fits none

◦ Need to balance centralization and agility

◦ Different APIs for different functions


API as a Domain Model


API as a workflow modelAffordances:◦ Action-oriented hyperlinks

◦ Transaction-oriented resources

◦ Navigational links to related resources

Workflow and affordances vary by service context.

Variations often require different APIs for different clients.


API as a Client ViewIt’s not necessarily just a domain model dump.

It’s the domain as the client sees it: Respect the Client’s vocabulary

Points of Divergence:◦ Property subsets◦ Flattening◦ Renaming◦ Change of perspective.◦ Reification: example, refund


Canonical Models and VariationsThere’s a common theme (canon)underneath these variations.

See how far you can go with canonical models and realizations:◦ Can we describe the theme and

variations separately?

◦ Can we model the variations as adaptations, augmentations of the theme?

◦ If so, maybe we don’t needAPI-specific data types

36COPYRIGHT © 2014, MODELSOLV, INC. | ALL RIGHTS RESERVED.

Realization: Property Subset


API requests or responses may only need a subset of properties defined in the canonical model.

Realization model may specify a list of included properties.

Realization: Perspective


Message and resource structures project different views from the same logical data model

Canonical model should support bi-directional references.

Realization model should allow embedded or linked representations.

Realization: Metadata


Business Information Model

Account ID

Balance

Margin

Status

Account

Party ID

Name

Party1

0..n

Data Aspects

· Deltas

· Data Source

· Data Security

· Explicit Null Values

...

Message Structure

<party dataSource=“MSDB”>

<partyId>123</partyId>

<partyName xsi:nil=“true” nullValue=“Not Available” />

<accounts>

<account dataSource=“A2” transType=“insert”>

<accountId>XYZ</accountId>

<balance xsi:nil=“true” isRestricted=“true” />

…

</account>

…

</accounts>

</party>

APIs may need to augment essential data with descriptive metadata.

Data aspects are cross-cutting concerns that may be woven together with canonical data as part of the interface realization.

Realization: Contextual Constraints

<=$10MMAsset

Class = Bond


TradeServices may have specific constraints that are not intrinsic to the data definitions.

Realization model may specify constraints on requests or responses. Constraints may take different forms: range, subtype, logical expression, etc.

Conclusion: Get involved!We need your modeling skills.

Done right, new API initiatives can have huge positive impact.

Done poorly, they can actually make things worse.◦ Net Negative Contributing API (NNCAPI)

◦ … or Convergence-Resistant API ;-)


ResourcesREST:◦ Basic REST in action:

http://www.youtube.com/watch?v=7YcW25PHnAA

◦ HTTP and REST:http://code.tutsplus.com/tutorials/a-beginners-guide-to-http-and-rest--net-16340

◦ REST API Design Principles:http://www.infoq.com/articles/rest-introduction

◦ REST Idioms and Patterns:http://www.narwhl.com/

API Blogs◦ http://apievangelist.com

◦ http://programmableweb.com


http://www.youtube.com/watch?v=7YcW25PHnAA

http://code.tutsplus.com/tutorials/a-beginners-guide-to-http-and-rest--net-16340

http://www.infoq.com/articles/rest-introduction

http://www.narwhl.com/

http://apievangelist.com/

http://programmableweb.com/

Resources (continued)Canonical Models:◦ http://www.slideshare.net/TedEpstein/canonical-modeling-for-api-interop

◦ http://www.digitalml.com/resources/Service_Design_Forum

API Description Languages◦ http://swagger.io/

◦ http://apiblueprint.org/

◦ http://raml.org/

◦ http://modelsolv.com/reprezen


http://www.slideshare.net/TedEpstein/canonical-modeling-for-api-interop

http://www.digitalml.com/resources/Service_Design_Forum

http://swagger.io/

http://apiblueprint.org/

http://raml.org/

http://modelsolv.com/reprezen

QuestionsTHANK YOU!

[email protected]


data modeling in the api economy

Technology

data modelers

data conduits

data security

disparate data

dataoriented apis

data modeling zone

data quality constraints

rest standards