dropwizard with mongodb and google cloud
DESCRIPTION
A one day prototype I completed for Apmasphere.com using Dropwizard plus MongoDB deployed onto Google Compute Engine.TRANSCRIPT
BILL APIONE DAY PROTOTYPE USING
, ANDDROPWIZARD MONGODBGOOGLE CLOUD
Yun Zhi Lin +YunZhiLin @YunZhiLin
with Ray King+RayKing
ABOUT - YUN ZHI LINComputer Science @UNSW, Applied Finance @MacquarieJoined a startup in 2002, then moved into Banking ITMoonlights as a Web 2.0 (nosql, json, mobile) enthusiastLoves Java without bloated frameworks or appserversFather of one chubby "michelin" baby son
SPECEnd to end prototype of a Bill Aggregator that mimics
vendor API. See below for detail specifications.
DATA MODELProviders - bill provider
Premises - location being provisioned for
Subscriptions - subscribe to bill by account/reference #
Bill - matching subscriptions trigger notifications
Notifications - Sent to subscription.callbackUrl withUrlForBillRetrieval
FK: List(Subscriptions)
FK: List(Subscriptions)
FK: Provider.id, Premise.id, Bill.accountNumber, List(Bill.id, Status)
FK: Provider.id, Subscription.referenceNumber
FK: Provider, Premise, Subscription
ACTORSProducer - API hosted on Google Compute EngineAPI Operations - Insert, List, Get, Remove, Path on allentities, except Notification and Bill.
Consumer - Use Sample Data - JSON files provided
Postman HTTP client
REQUIREMENTS1. HTTP transport2. JSON payload3. Use compute engine4. Authentication5. API versioning6. Ops-friendly7. Data Validation8. Batch considerations
DESIGN
FRAMEWORKS CHOICENode.js - full stack JSON, but lacking expertisePlay! Framework - too complex for this exerciseDropwizard - simple, war-less and proven
DATABASE CHOICEGoogle Cloud Datastore - lacking expertiseMongoDB - perfect for JSON
DROPWIZARD + MONGODB = MEETREQUIREMENTS
1. HTTP - Jersey RESTful web services2. JSON - Jackson with Mongo-Jack, fastest Java parser3. Use compute engine - Both API jar and DB hosted on GCE4. Authentication - OAuth2 or custom Providers5. API versioning - annotation based url mappings6. Ops-friendly - healchecks and Codahale metrics7. Data Validation - Hibernate Validatior8. Batch considerations - use JSON lists
PLANDue to the time contraint, it's important to decide on the
approach taken to plan out a successfull prototype:
Start to End - build each component fully one by one, canonly partially demonstrate the storyMinimum Viable Product - selectively build only criticaloperations for all components. So that together they tell afull story
SCOPE ADJUSTMENTAuthentication - excludedAPI versioning - excludedBatch - simplified to single JSON parsingQueuing - another consideration for the real world,excludedNotification - manually triggered.
Only the following Operations demonstrate the coreprocesses of subscription, notification and bill retrieval:
FINAL USE CASE
1. create a subscriptoin
2. Manually trigger notification, where UrlForBillRetrieval =subscription.callbackURL + billID
3. retrieve a Bill
PUT /subscription/
GET /subscription/trigger/{subscriptoinID}/{billID}/
GET /bill/?billID={billID}
IMPLEMENTATION1. Build Tool2. Configuration3. MongoDB4. Healthcheck5. JSON POJOs6. Resources7. Application Glue
BUILD TOOLGradle combines the best of Ant and Maven
repositories { mavenLocal() mavenCentral()}project.ext { dropwizardVersion = '0.7.0-SNAPSHOT' mongojackVersion = '2.0.0-RC5' dropWizardConfig = './src/main/resources/billproto.yml'}dependencies { compile 'io.dropwizard:dropwizard-core:' + dropwizardVersion compile 'org.mongojack:mongojack:' + mongojackVersion testCompile group: 'junit', name: 'junit', version: '4.11'} run { args 'server', dropWizardConfig}
CONFIGURATIONYAML
POJO
mongo: host: ds039487.mongolab.com port: 39487 db: tycoon-mongo user: #### password: ####
public class MongoConfiguration { @NotNull public String host; @Min(1) @Max(65535) @NotNull public int port; @NotNull public String db; @NotNull public String user; @NotNull public String password;}
MONGODBHosting - GCE optionConnection
JacksonDBCollection
DB Import (must be list of single line json files)
MongLab
mongo = new Mongo(mongoConfig.host, mongoConfig.port);db = mongo.getDB(mongoConfig.db);db.authenticate(mongoConfig.user, mongoConfig.password.toCharArray());
bills = wrap(db.getCollection("bill"), Bill.class, String.class);dbCursor = bills.find().is("_id", billID);if (dbCursor.hasNext()) { Bill result = cursor.next();}
mongoimport --host --username --password --db --collection --file
HEALTHCHECKCode
Healthy
Not healthy:
@Overrideprotected Result check() throws Exception { mongo.getDB(dbname).getCollectionNames(); return Result.healthy();}
{"MongoHealthCheck":{"healthy":true},"deadlocks":{"healthy":true}}
{"MongoHealthCheck":{"healthy":false, "message":"not authorized for query on tycoon-mongo.system.namespaces", "error":{"message":"not authorized for query on tycoon-mongo.system.namespaces" "stack":[...] "deadlocks":{"healthy":true}}
JSON POJOSUse . Then apply simple
annotations for Jackson and Hibernate Validatior.http://www.jsonschema2pojo.org/
@JsonInclude(JsonInclude.Include.NON_NULL)public class Notification {
@Id public String notificationID;
@NotNull public String subscriptionID;
@NotNull public String notificationURLForBillDataRetrival;
@NotNull public String notificationURLForBillImageRetrival;
}
NOTE: Use @Id for UUID, not @ObjectId
RESOURCESAnnotate opertion, metrics, url path and output format.
Jackson automatically parse POJO into specified mediaType.
@GET@Timed@Path("trigger/{subscriptionID}/{billID}")@Produces(MediaType.APPLICATION_JSON)public Notification triggerNotification( @PathParam("subscriptionID") String subscriptionID, @PathParam("billID") String billID) { ... notification = new Notification(); notification.notificationID = UUID.randomUUID().toString(); notification.subscriptionID = subscriptionID; notification.notificationURLForBillDataRetrival = subscription.subscriptionCallBackURL + "?billID=" + billID; return notification;}
APPLICATION GLUEPutting all of the above together
@Overridepublic void run( BillprotoConfiguration configuration, Environment environment) throws Exception {
MongoManaged mongoManaged = new MongoManaged(configuration.mongo); environment.lifecycle().manage(mongoManaged); environment.healthChecks().register( "MongoHealthCheck", new MongoHealthCheck(mongoManaged));
environment.jersey().register(new SubscriptionResource(mongoManaged)); environment.jersey().register(new BillResource(mongoManaged));}
DEPLOY1. Install Java (defaults to OpenJDK)
2. Open firewall ports 8080 and 8081
3. Copy file to server
4. Run fatjar
sudo apt-get install java7-runtime-headless
gcutil addfirewall rest --description="http" --allowed="tcp:8080gcutil addfirewall admin --description="Iadmin" --allowed="tcp:8081"
gcutil --project={project-id} push {instance-name} {local-file} {remote-target-path}
java -jar billproto-0.2-fat.jar server billproto.yml
TESTINGREST URLS
Subscription: Trigger:
Bill: Healthcheck:
162.222.183.244:8080/subscription/
162.222.183.244:8080/subscription/{subscriptionID}/{billID}162.222.183.244:8080/bill/?billID={billID}
162.222.183.244:8081
CONCLUSIONBy building just the bare minimum operations in each
components that work well together, we were able to coverthe core user story and better assess our API feasibility.
We also managed to test out a new framework and built atemplate that can be fleshed out for future applications.
LINKS AND CREDITS
(Private Repo) built with
Managed by and hosted on .
dropwizard.iomongojack.orggradle.orgSource codeThis Slide reveal.js Export to PDF
bower GitHub Pages
QUESTIONS