API and Big data Solution Patterns

Srinath Perera Senior Software Architect

About  WSO2  

•  Providing  the  only  complete  open  source  componen:zed  cloud  pla=orm  

–  Dedicated  to  removing  all  the  stumbling  blocks  to  enterprise  agility  

–  Enabling  you  to  focus  on  business  logic  and  business  value  

•  Recognized  by  leading  analyst  firms  as  visionaries  and  leaders  –  Gartner  cites  WSO2  as  visionaries  in  all  3  categories  of    

applica:on  infrastructure  –  Forrester  places  WSO2  in  top  2  for  API  Management  

•  Global  corpora:on  with  offices  in  USA,  UK  &  Sri  Lanka  –  200+  employees  and  growing  

•  Business  model  of  selling  comprehensive  support  &  maintenance  for  our  products  

 

150+  globally  posi:oned  support  customers  

Outline •  What  is  API  Management  

•  What  is  Big  Data?  

•  API  manager  +  Bigdata  solu:on  

•  API  +  Big  data  PaVerns  

•  Conclusion  

Photo  by  John  Trainoron  Flickr  hVp://www.flickr.com/photos/trainor/2902023575/,  Licensed  under  CC  

APIs

•  An API is a business capability delivered over the Internet to internal or external consumers –  Network accessible function –  Available using standard web protocols –  With well-defined interfaces –  Designed for access by third-parties

•  A Managed API is: –  Actively advertised and subscribe-able –  Available with SLAs –  Secured, authenticated, authorized and protected –  Monitored and monetized with analytics

Typical Use Cases

•  Expose APIs for public consumption –  Extend your business through APIs. •  Expose APIs for internal consumption – Manage the APIs used in internal applications – Detect Usage Patterns –  Internal Monetization

•  Control Access to Cloud Services – Manage and Secure access from internal

applications to cloud services (SalesForce, Google Apps, etc.)

WSO2 API Management Platform

WSO2’s API Management Vision •  Create APIs –  WSO2 Application Server, Data Services Server and ESB

•  Find and subscribe/buy APIs –  API Store and Governance

•  Manage, secure and protect APIs –  API Management and Gateway

•  Monitor and Monetize APIs –  API Monitoring and Analytics

•  Develop, host and run API-based applications in a Platform-as-a-Service –  WSO2 Stratos and Cloud Development Platform –  API Cloud

API Manager Components

API Gateway Processing Flow

API Manager Product and Platform

What is Big data?

•  There is lot of data available –  E.g. Internet of things

•  We have computing power

•  We have technology •  Goal To know –  To Explain –  To predict

•  Challenge is the full lifecycle

Drivers of Big Data

Big Data Architecture

Collecting Data •  Data collected at sensors and sent to big data

system via events or flat files •  Event Streams: we name the events by its

content/ originator

•  Get  data  through    –  Point  to  Point  –  Event  Bus  (WSO2  MB)  

•  E.g.  Data  bridge  (come  with  WSO2  CEP/  BAM)  – a  thrib  based  transport  that  do  about  400k  events/  sec  

Storing Data •  Historically we used databases

–  Scale is a challenge: replication, sharding

•  Scalable options –  NoSQL (Cassandra, Hbase) [If

data is structured] •  Column families Gaining Ground

–  Distributed file systems (e.g. HDFS) [If data is unstructured]

•  New SQL –  In Memory computing, VoltDB

•  Specialized data structures –  Graph Databases, Data structure

servers hVp://www.flickr.com/photos/keso/363133967/  

Making Sense of Data •  To know (what happened?) –  Basic analytics +

visualizations –  Interactive drill down –  E.g. MapReduce, Hive, CEP

•  To explain (why) –  Data mining, classifications,

building models, clustering (e.g. Mahout, Map Reduce)

•  To forecast –  Neural networks, decision

models (e.g. Mahout)

WSO2 Big Data Story

API Gateway + Big Data

API + Big Data Patterns •  Big Data as API enabler

–  Monitor and control: API analytics for an organization

–  Analytics: Billing and Metering –  API Store (Market Place)

•  Targeted Marketing (Recommendations, Deals, Search)

•  Context sensitive prizing

–  Deployment optimization –  Governance: Planning and Proactive

action •  APIs as a Big Data Enabler

–  Easy way to find and get access to data –  Data Market Place –  API Façade for exposing Analytics/ Data –  Controlled delegation of data

hVp://www.flickr.com/photos/ooohoooh/1350774047/  

Pattern 1: Monitor and Control

Monitor and Control (Contd.) •  API Manager as single entry point to come in and go

out •  Capture data – client, what API, when? $$ revenue .. •  Analyze and Monitor KPIs

–  CIO Dashboard –  Alerts

•  React to trends faster •  Greater control

–  Audit –  Subscriptions policies –  Usage tiers

•  Policy enforcement –  Make sure everything is managed

Example: Business value driven Org •  Customer facing site is a

composite (e.g. like Gmail page) •  Each widget back by API or APIs •  Each widget added removed in

runtime •  Widget shown based on business

value •  Each API owner get resource

compared to business value •  We can use API management as

the tool to realize this

Pattern 2: Billing and Metering •  API store detect all access •  Contains information about

subscriptions, amount of data transferred etc.

•  We can use this to –  Meter, control and optimize –  Enforce usage tiers (e.g. Quota

via CEP)

•  Generate Billing and reports •  Supported by WSO2 API

manager now

Pattern 3: API Store (Marketplace)

Pattern 4: Deployment Optimization

App Factory - Applications Management

App Factory (Contd.) •  Applications Management –  Creation –  Continuous Build / Deploy –  Promotion across environments

•  Team Management –  Add developers, QA engineers and DevOps for your

application •  Bugs/Features Tracking •  Integrated with API Store •  Activity Dashboards •  Built on top of Jaggery framework •  Skinnable

Pattern 5:API lifecycle Governance •  Take control of your API life cycle

(develop, test, production) with WSO2 App Factory

•  You can apply analytics to your API lifecycle –  How long does it take to develop and

API? –  How often are they updated? –  What are the dependencies of my

API? –  What are most effective versions? –  What steps takes most time? –  What APIs are actively developed? –  What one have more issues, how fast

are they get fixed?

Pattern 6: Find datasets

•  Most big data apps need Good data

•  Hard to find good data –  No good way to find –  Hard to access –  No clear way to buy

•  APIs is a great way to find useful data

•  Also to provide controlled access

e.g. Andy Stanford-Clark used public data about cruisers to track where availability.

Pattern 7: Data Marketplace •  Find, (buy), use data •  Great way to monetize the

output of big data projects –  Sell the analyzed traffic data –  Sell stock quote analytics

•  API subscribe model can help •  Usage tiers can be based on

frequency or resolution of data

Pattern 8: Controlled Data Delegation •  Only expose the part of your data set of a processing

without exposing credentials of the database to the user

Big data in API Store

APIs in Big data System

Usecase •  Trigger an alert when new version is 10% more

slower than the older versions –  Expose the API through API store –  Configure API manager with BAM location –  BAM integration will pump stats (api, version, latency,

time) –  Write a BAM query to calculate the latency through

time and store it in a DB –  Write a CEP query to detect if new version is slower

Define eventTable APIStatHistory .. Select from APIStat as s join APIStatHistory as h on (h.latency -

s.latency)<=h.latency*.1

Conclusion

•  APIs and Big Data has received much attentions

•  We looked at WSO2 offerings in both cases – WSO2 API Manager, WSO2 App Factory – WSO2 BAM, WSO2 CEP

•  We looked at some patterns on how both can work together

•  We looked at how we can implement them

Engage  with  WSO2  

•  Helping  you  get  the  most  out  of  your  deployments  •  From  project  evalua:on  and  incep:on  to  development  and  going  into  produc:on,  WSO2  is  your  partner  in  ensuring  100%  project  success  

 

Questions?