lightblue: enterprise data services in the cloud with mongodb
DESCRIPTION
Overview of project lightblue, an open source project for enabling deployment of enterprise data services (SOA) to the cloud. Lightblue has a pluggable software architecture and supports MongoDB persistence at this time with more support coming soon (such as RDBMS). lightblue on github: https://github.com/lightblue-platform/lightblue Free books on gitbook.io: http://jewzaam.gitbooks.io/lightblue/ http://jewzaam.gitbooks.io/lightblue-user-guide/ http://jewzaam.gitbooks.io/lightblue-developer-manual/TRANSCRIPT
lightblue:
Naveen MalikPrincipal Software Applications Engineer, Red HatJuly 24, 2014
Enterprise Data Services in the Cloud with MongoDB
Naveen Malik http://goo.gl/f7S4PA
Where can I get this presentation?
Also in footer...
Naveen Malik http://goo.gl/f7S4PA
Introduction
● Goals:o Share project lightblue.o Answer your questions.o Listen to your ideas.
● Please interrupt for questions, comments, ideas, etc.
Naveen Malik http://goo.gl/f7S4PA
What is lightblue?Cloud focused data services with dynamic querying, versioned schemas, and robust security.
Naveen Malik http://goo.gl/f7S4PA
Why Open Source?
● Code Qualityo More eyes, more ideas, better code.
● Better Architectureo More mindful of separation of concerns
● Giving back!o Not a unique problem, others can benefit and contribute.o Contributors bring ideas, testers, developers, and more!
Naveen Malik http://goo.gl/f7S4PA
Why was lightblue created?
● Provide clients standard API for any CRUD operations.● Enable deployment anywhere with robust security.● Rapid response to changing business needs.● Stability for existing consumers.
Naveen Malik http://goo.gl/f7S4PA
How does lightblue benefit developers?
● Applications have a robust lightblue API● Versioned metadata allows changes without breaking
existing applicationso clients see their own version of datao different clients operate on same object with different versionso backwards compatibility of minor changes guaranteed
Naveen Malik http://goo.gl/f7S4PA
How does lightblue benefit operations?
● Designed to be deployed anywhereo AWSo Rackspaceo OpenShifto Embedded as a library in other applications
● Flexible component architectureo Deploy and scale components independently
● Latency and fault tolerance● Robust security
Naveen Malik http://goo.gl/f7S4PA
How does lightblue benefit SOA teams?
Naveen Malik http://goo.gl/f7S4PA
What about support and development tools?
Naveen Malik http://goo.gl/f7S4PA
Why MongoDB?
● Lightweight and flexible● Scale horizontally by adding nodes● Replication out of the box● Ability to shard (partition) out of the box
First controller implementation! RDBMS coming later.
Naveen Malik http://goo.gl/f7S4PA
How does it work?
● JSON in● JSON to BSON● BSON to JSON● JSON out
Naveen Malik http://goo.gl/f7S4PA
Metadata
● Entity Infoo not versionedo persistence informationo indexes, enumerations
● Schemao versionedo fieldso constraintso access rights
Naveen Malik http://goo.gl/f7S4PA
User Metadata - Entity Info
"entityInfo" : {"name": "user","enums" : [{
"name": "site_type_enum", "values": [ "billing", "marketing", "service", "shipping" ] }],
"datastore": { "backend":"mongo", "datasource": "mongo", "collection": "user" }
} }
Naveen Malik http://goo.gl/f7S4PA
User Metadata - Schema
"schema" : {"name" : "user","version": { "value": "1.0.0", "changelog": "Test version" },"status": {"value": "active"},"access" : {
"insert": ["anyone"], "find":["anyone"], "update":["anyone"], "delete":["anyone"]
},"fields": {...}
}
Naveen Malik http://goo.gl/f7S4PA
User Metadata - Fields and Objects
"fields": { "object_type": { "type": "string" }, "login": { "type": "string", "constraints": { "maxLength": 64, "minLength": 1, "required": true } },...
Naveen Malik http://goo.gl/f7S4PA
Should we risk a live demo?
Setup● OpenShift Online - https://www.openshift.com/● Small jbosseap-6 gear with mongodb-2.4 cartridge● Setup following https://github.com/lightblue-platform/openshift-lightblue-all
List all entities:● curl http://services-lightblue.rhcloud.com/rest/metadata/
All versions for ‘country’:● curl http://services-lightblue.rhcloud.com/rest/metadata/country/
Details of country with iso2Code of ‘US’:● curl http://services-lightblue.rhcloud.com/rest/data/find/country/1.0.0?Q=iso2Code:US
Naveen Malik http://goo.gl/f7S4PA
How powerful is lightblue’s query API?
● Queries:o “field”=”value”, “field1”=”field2”o $and, $or, $in, $nino Regular expressionso Array element searches
● Projections: Ability to get what you wanto Include some fieldso Exclude some fieldso Array rangeso Return matching array elements
Naveen Malik http://goo.gl/f7S4PA
How does lightblue keep my data safe?
● Recommend encryption in transit and at resto Container can terminate SSL from cliento Support SSL communication with MongoDB
● Secure by designo Ability to control access to data at entity and field levelo Data access controlled down to individual fields
● Authentication & authorization as plugino JBoss EAP 6 with PicketLink and Login Moduleso Authentication with SAML 2.0 (people) and Certificates (systems)o Authorization with LDAP
Naveen Malik http://goo.gl/f7S4PA
What about MongoDB security?
● Restrict access to database!o ACL, Security Group, iptables, firewall, etc.
● Require SSL!o SSL is enabled in community edition 2.6.0+o Require certificate from client.o http://docs.mongodb.org/manual/tutorial/configure-ssl/
Naveen Malik http://goo.gl/f7S4PA
Logical Reference Architecture
Naveen Malik http://goo.gl/f7S4PA
Provisioning
● CloudFormation - template for each node type● Cumulus - orchestration of node creation● Puppet - configuration of nodes after creation
o Our module is on github!o https://github.com/lightblue-platform/lightblue-puppet
Naveen Malik http://goo.gl/f7S4PA
Monitoring MongoDB
● Mongo Management Serviceo https://mms.mongodb.com/o Agent connects to mongo databaseo Can be shared across environmentso Supports secure connections over SSL
● Further readingo http://docs.mongodb.org/manual/administration/monitoring/
Naveen Malik http://goo.gl/f7S4PA
Lessons Learned
Nothing is perfect…● JSON
o Date isn’t done well in JSON, we decided on a standard.o BigDecimal and BigInteger support.o Solution: solve with standards and in JSON conversion
● MongoDBo No transaction support.o Ad hoc queries against large data sets o Solution: implement other controllers: RDBMS
Naveen Malik http://goo.gl/f7S4PA
What’s next for lightblue?
● Associations (joins) between entitieso 1-1, 1-n, n-n associations defined in metadatao Different cascading options for insert/update/deleteo Ad-hoc associations in queries
● Asynchronous processingo Call, receive a handle, don’t wait for response to completeo Check handle to get status
Naveen Malik http://goo.gl/f7S4PA
Can I move existing data to lightblue?
YES!
Expect pattern for migration to be published soon.And some utilities to make it easier...
Naveen Malik http://goo.gl/f7S4PA
Want more information?
● Sourceo https://github.com/lightblue-platform/lightblue
● Documentationo See gitbook links in lightblue README for overview, user guide, and
developer manual.● IRC: Freenode / #lightblue● Forums:
o Developer Forum o User Forum
Naveen Malik http://goo.gl/f7S4PA
What questions can I answer?
● Thoughts you would like to share?