wed 1630 greene_robert_color

28
Versant Innovation Leveraging your Knowledge of ORM Towards Performance-based NoSQL Technology Versant Corporation U.S. Headquarters 255 Shoreline Dr. Suite 450, Redwood City, CA 94065 www.versant.com | 650-232-2400

Upload: dataversity

Post on 20-Aug-2015

429 views

Category:

Technology


0 download

TRANSCRIPT

Page 1: Wed 1630 greene_robert_color

Versant Innovation Leveraging your Knowledge of ORM Towards Performance-based NoSQL Technology

Versant Corporation U.S. Headquarters 255 Shoreline Dr. Suite 450, Redwood City, CA 94065 www.versant.com | 650-232-2400

Page 2: Wed 1630 greene_robert_color

NoSQL at it’s Core

Pole-Position – Overview

About the Code

Circuits Description

RDB JPA Code

MongoDB Code

Versant JPA Code

Results – Winners of the Race

Developer Challenge

Overview

Page 3: Wed 1630 greene_robert_color

NoSQL at its Core

Page 4: Wed 1630 greene_robert_color

A Shift In Application Architecture

• Google – Soft-Schema • IBM – Schema-Less

Inefficient CPU destroying

Mapping

Excessive Repetitive data

movement and JOIN calculation

Page 5: Wed 1630 greene_robert_color

Why the Shift is Needed

• Think about it – How Often do Relations Change? – Blog : BlogEntry , Order : OrderItem , You : Friend

Stop Banging Your Head on the Relational Wall Relations Rarely Change, Stop Recalculating Them You don’t need ALL your Data, you can distribute

Page 6: Wed 1630 greene_robert_color

Measured Value

How NoSQL performance stacks up to Relational

Page 7: Wed 1630 greene_robert_color

PolePosition Performance Benchmark

• Established in 2003 – NoSQL -vs- ORM

• Code Complexity Circuits – Flat Objects – Graphs – Inherited Objects – Collections

• Data Operations – CRUD – Concurrency

• Scalability

Page 8: Wed 1630 greene_robert_color

Contenders | Methodology

• Contenders – ORM

• Hibernate – MySQL / Postgres • OpenJPA – MySQL / Postgres (other RDB, cannot publish – same

basic results) – NoSQL

• MongoDB • Versant Database Engine

• Methodology – External RDB Experts, Internal NoSQL Experts – Open Source Benchmark Code for all contenders

Page 9: Wed 1630 greene_robert_color

About the Code

• ORM - Common Code Base – Hibernate – MySQL / Postgres – OpenJPA – MySQL / Postgres – MongoDB – Versant

Page 10: Wed 1630 greene_robert_color

Circuit Structure • Simulate CRUD on embedded graph of different Classes. • Class model - Complex Circuit: class Holder0 { String _name; List <Holder0> _children; Holder0[] _array; } class Holder1 extends Holder0 { int _i1; } class Holder2 extends Holder1 { int _i2 <indexed>; } ...class HolderN extends…..

Page 11: Wed 1630 greene_robert_color

Hibernate JPA Code

• Write – Generates Holder’s to user spec depth ComplexHolder cp = new ComplexHolder( ); em.makePersistent(cp);

• Read – Access root of graph and traverse cp = em.find( ComplexHolder.class, id ); cp.getChildren().touch();

• Query String query = "from org.polepos.teams.hibernate.data.Holder2 where

i2=" + currentInt; Iterator it = em.iterate(query);

• Delete – Deletes Graph - cascading operation

Page 12: Wed 1630 greene_robert_color

ORM JPA Mapping

• - <hibernate-mapping package="org.polepos.teams.hibernate.data" default-cascade="none" default-access="property" default-lazy="true" auto-import="true"> • - <class name="ComplexHolder0" table="tComplexHolderNew0" polymorphism="implicit" mutable="true" dynamic-update="false" dynamic-insert="false" select-before-update="false"

optimistic-lock="version"> • <id name="id" column="fid" type="long" /> • - <discriminator type="string" not-null="true" force="false" insert="true"> • <column name="DISCRIMINATOR" /> • </discriminator> • <property name="name" column="fname" type="string" unique="false" optimistic-lock="true" lazy="false" generated="never" /> • - <list name="children" access="field" cascade="all" inverse="false" mutable="true" optimistic-lock="true" embed-xml="true"> • <key column="parentId" on-delete="noaction" /> • <index column="elementIndex" /> • <many-to-many class="ComplexHolder0" embed-xml="true" not-found="exception" unique="false" /> • </list> • - <array name="array" access="field" cascade="all" inverse="false" mutable="true" optimistic-lock="true" embed-xml="true"> • <key column="parentId" on-delete="noaction" /> • <index column="elementIndex" /> • <many-to-many class="ComplexHolder0" embed-xml="true" not-found="exception" unique="false" /> • </array> • - <subclass name="ComplexHolder1" discriminator-value="D" dynamic-update="false" dynamic-insert="false" select-before-update="false"> • <property name="i1" column="i1" type="int" unique="false" optimistic-lock="true" lazy="false" generated="never" /> • - <subclass name="ComplexHolder2" discriminator-value="E" dynamic-update="false" dynamic-insert="false" select-before-update="false"> • <property name="i2" column="i2" type="int" index="i2_idx" unique="false" optimistic-lock="true" lazy="false" generated="never" /> • - <subclass name="ComplexHolder3" discriminator-value="F" dynamic-update="false" dynamic-insert="false" select-before-update="false"> • <property name="i3" column="i3" type="int" unique="false" optimistic-lock="true" lazy="false" generated="never" /> • - <subclass name="ComplexHolder4" discriminator-value="G" dynamic-update="false" dynamic-insert="false" select-before-update="false"> • <property name="i4" column="i4" type="int" unique="false" optimistic-lock="true" lazy="false" generated="never" /> • </subclass> • </subclass> • </subclass> • </subclass> • </class> • </hibernate-mapping>

XML Mapping File for each Persistent Class ( 11ea Files )

Page 13: Wed 1630 greene_robert_color

MongoDB Code private void storeData(Mongo mongo) { collection(mongo).insert(new BasicDBObject("test", "object")); } public ComplexHolder0 convertFromDocument(DBObject data, DeserializationOptions deserializationOption) { ComplexHolder0 instance = createInstance(getAsString(data, TYPE_ATTRIBUTE)); instance.setName(getAsString(data, NAME_ATTRIBUTE)); if (null != data.get(CHILDREN)) { instance.setChildren(fromMongoObjectToList(getAsList(data, CHILDREN), deserializationOption)); } if (null != data.get(ARRAY)) { final List<ComplexHolder0> arrayContent = fromMongoObjectToList(getAsList(data, ARRAY),

deserializationOption); instance.setArray(arrayContent.toArray(new ComplexHolder0[arrayContent.size()])); } readAttributes(data, instance); return instance; }

Page 14: Wed 1630 greene_robert_color

MongoDB Mapping private static void writeSubTypeAttributes(ComplexHolder0 holder, BasicDBObject

dataStorage) { if (holder instanceof ComplexHolder1) { dataStorage.put(FIELD_I1, ((ComplexHolder1) holder)._i1); } if (holder instanceof ComplexHolder2) { dataStorage.put(FIELD_I2, ((ComplexHolder2) holder)._i2); } private static void readAttributes(DBObject dataStorage, ComplexHolder0 holder) { if (holder instanceof ComplexHolder1) { ((ComplexHolder1) holder)._i1 = (Integer) dataStorage.get(FIELD_I1); } if (holder instanceof ComplexHolder2) { ((ComplexHolder2) holder)._i2 = (Integer) dataStorage.get(FIELD_I2); }

Page 15: Wed 1630 greene_robert_color

MongoDB Mapping class Serialisation { static final String TYPE_ATTRIBUTE = "_t"; private static final String PACKAGE_NAME = ComplexHolder0.class.getPackage().getName(); static final String NAME_ATTRIBUTE = "name"; static final String CHILDREN = "children"; static final String ARRAY = "array"; static final String REFERENCE_TO_ORIGINAL_DOCUMENT = "_refToOriginal"; static final String FIELD_I1 = "_i1"; static final String FIELD_I2 = "_i2"; static final String FIELD_I3 = "_i3"; static final String FIELD_I4 = "_i4"; private final OneArgFunction<BasicDBObject, DBRef> refCreator; Serialisation(OneArgFunction<BasicDBObject, DBRef> refCreator) { this.refCreator = refCreator; } public static Serialisation create(OneArgFunction<BasicDBObject, DBRef> refCreator) { if (null == refCreator) { throw new ArgumentNullException("requires a reference creator"); } return new Serialisation(refCreator); } public static OneArgFunction<BasicDBObject, DBRef> createReferenceCreator(final DBCollection collection) { if (null == collection) { throw new ArgumentNullException("requires a collection"); } return new OneArgFunction<BasicDBObject, DBRef>() { @Override public DBRef invoke(BasicDBObject basicDBObject) { final DB db = collection.getDB(); final Object id = basicDBObject.get("_id"); if (null == id) { throw new IllegalStateException("Expected an '_id' on the object"); } return new DBRef(db, collection.getName(), id); } }; }

public SerialisationResult convertToDocument(ComplexHolder0 holder) { List<BasicDBObject> holder2Objects = new ArrayList<BasicDBObject>(); BasicDBObject document = convertToDocument(holder, holder2Objects); return new SerialisationResult(document, holder2Objects); } public ComplexHolder0 convertFromDocument(DBObject data) { return convertFromDocument(data, DeserializationOptions.FULL_DESERIALISATION); } public ComplexHolder0 convertFromDocument(DBObject data, DeserializationOptions deserializationOption) { ComplexHolder0 instance = createInstance(getAsString(data, TYPE_ATTRIBUTE)); instance.setName(getAsString(data, NAME_ATTRIBUTE)); if (null != data.get(CHILDREN)) { instance.setChildren(fromMongoObjectToList(getAsList(data, CHILDREN), deserializationOption)); } if (null != data.get(ARRAY)) { final List<ComplexHolder0> arrayContent = fromMongoObjectToList(getAsList(data, ARRAY), deserializationOption); instance.setArray(arrayContent.toArray(new ComplexHolder0[arrayContent.size()])); } readAttributes(data, instance); return instance; } private BasicDBObject convertToDocument(ComplexHolder0 holder, List<BasicDBObject> referencedDocumentsCollector) { BasicDBObject dbObject = createDocumentWithAttributesOnly(holder); dbObject.put(CHILDREN, toMongoObject(holder.getChildren(), referencedDocumentsCollector)); if (null != holder.getArray()) { dbObject.put(ARRAY, toMongoObject(asList(holder.getArray()), referencedDocumentsCollector)); } return dbObject; } private BasicDBObject createDocumentWithAttributesOnly(ComplexHolder0 holder) { BasicDBObject dbObject = new BasicDBObject("_id", new ObjectId()); dbObject.put(NAME_ATTRIBUTE, holder.getName()); dbObject.put(TYPE_ATTRIBUTE, holder.getClass().getSimpleName()); writeSubTypeAttributes(holder, dbObject); return dbObject; } public List<ComplexHolder0> fromMongoObjectToList(List data, DeserializationOptions deserializationOption) {

List<ComplexHolder0> objects = new ArrayList<ComplexHolder0>(data.size()); for (Object o : data) { objects.add(restoreDocumentOrReference(o, deserializationOption)); } return objects; } private ComplexHolder0 restoreDocumentOrReference(Object o, DeserializationOptions deserializationOption) { DBObject dbObj = (DBObject) o; if (null != dbObj.get(REFERENCE_TO_ORIGINAL_DOCUMENT)) { return deserializationOption.deserialize(this, dbObj); } else { return convertFromDocument(dbObj, deserializationOption); } } private static List getAsList(DBObject data, String attributeName) { return (List) data.get(attributeName); } private static String getAsString(DBObject data, String attribute) { return (String) data.get(attribute); } private static ComplexHolder0 createInstance(String className) { try { return (ComplexHolder0) Thread.currentThread().getContextClassLoader() .loadClass(PACKAGE_NAME + "." + className).newInstance(); } catch (Exception e) { throw rethrow(e); } } private static void writeSubTypeAttributes(ComplexHolder0 holder, BasicDBObject dataStorage) { if (holder instanceof ComplexHolder1) { dataStorage.put(FIELD_I1, ((ComplexHolder1) holder)._i1); } if (holder instanceof ComplexHolder2) { dataStorage.put(FIELD_I2, ((ComplexHolder2) holder)._i2); } if (holder instanceof ComplexHolder3) { dataStorage.put(FIELD_I3, ((ComplexHolder3) holder)._i3); } if (holder instanceof ComplexHolder4) {

} if (holder instanceof ComplexHolder4) { dataStorage.put(FIELD_I4, ((ComplexHolder4) holder)._i4); } } private static void readAttributes(DBObject dataStorage, if (holder } if (holder instanceof ComplexHolder2) { ((ComplexHolder2) holder)._i2 = (Integer) dataStorage.get(FIELD_I2); } if (holder instanceof ComplexHolder3) { ((ComplexHolder3) holder)._i3 = (Integer) dataStorage.get(FIELD_I3); } if (holder instanceof ComplexHolder4) { ((ComplexHolder4) holder)._i4 = (Integer) dataStorage.get(FIELD_I4); } } private List<Object> toMongoObject(List<ComplexHolder0> children,) { List<Object> objects = new ArrayList<Object>(children.size()); for (ComplexHolder0 child : children) { final BasicDBObject document = convertToDocument(child, if (isDocumentOnlyReferenced(child)) { referencedDocumentsCollector.add(document); DBObject copy = createDocumentWithAttributesOnly(child); copy.put(REFERENCE_TO_ORIGINAL_DOCUMENT, objects.add(copy); } else { objects.add(document); } } return objects; } private static boolean isDocumentOnlyReferenced(ComplexHolder0 child) { return child.getClass().equals(ComplexHolder2.class); }

Page 16: Wed 1630 greene_robert_color

Versant JPA Code

• Write – Generates Holder’s to user spec depth ComplexHolder cp = new ComplexHolder( ); em.makePersistent(cp);

• Read – Access root of graph and traverse cp = em.find( ComplexHolder.class, id ); cp.getChildren().touch();

• Query String query = "from org.polepos.teams.versant.data.Holder2 where i2="

+ currentInt; Iterator it = session.iterate(query);

• Delete – Deletes Graph – Cascading Operation

Page 17: Wed 1630 greene_robert_color

Versant JPA Mapping

<class name="ComplexHolder0"> <field name="name" /> <field name="array" /> <field name="children"> <collection element-type="ComplexHolder0"/> </field> </class>

single XML File with primitive declaration entries

Page 18: Wed 1630 greene_robert_color

Sh@t’s and Grins • JDBC Code JDBC StringBuilder sb = new StringBuilder(); sb.append("select " +

HOLDER_TABLE0 + ".id from " + HOLDER_TABLE0); sb.append(" INNER JOIN " + HOLDER_TABLES[0]); sb.append(" on " + HOLDER_TABLE0 + ".id = " + HOLDER_TABLES[0] + ".id "); sb.append(" INNER JOIN " + HOLDER_TABLES[1]); sb.append(" on " + HOLDER_TABLE0 + ".id = " + HOLDER_TABLES[1] + ".id "); sb.append(" LEFT OUTER JOIN " + HOLDER_TABLES[2]); sb.append(" on " + HOLDER_TABLE0 + ".id = " + HOLDER_TABLES[2] + ".id "); sb.append(" LEFT OUTER JOIN " + HOLDER_TABLES[3]); sb.append(" on " + HOLDER_TABLE0 + ".id = " + HOLDER_TABLES[3] + ".id "); sb.append(" where " + HOLDER_TABLES[1] + ".i2 = ?"); PreparedStatement stat = prepareStatement(sb.toString()); ResultSet resultSet stat.executeQuery();

Page 19: Wed 1630 greene_robert_color

PolePosition BenchMark Summary Results

Page 20: Wed 1630 greene_robert_color

FlatObject - Single Thread

Page 21: Wed 1630 greene_robert_color

Results – ComplexConcurrency @mongo - no safe mode

Page 22: Wed 1630 greene_robert_color

Results – QueryConcurrency @mongo – no safe mode

Page 23: Wed 1630 greene_robert_color

Results – InsertConcurrent @mongo – no safe mode

Page 24: Wed 1630 greene_robert_color

Results – ComplexConcurrency @mongo – safe mode

Page 25: Wed 1630 greene_robert_color

Results – QueryConcurrency @mongo – safe mode

Page 26: Wed 1630 greene_robert_color

Results – InsertConcurrent @mongo – safe mode

Page 27: Wed 1630 greene_robert_color

Conclusions

• Wake-up, smell the coffee – JOINs are not needed - unless adhoc analytics

• Take care of business, dump a load of code – Serialization is bad, data useful once structured – Mapping is Mapping, even when it’s not an ORM

• Get on your Bad Motor Scooter and Ride – NoSQL without Mapping rules the road

Get the Code: http://www.polepos.org

Page 28: Wed 1630 greene_robert_color

Contact

Robert Greene Vice President, Technology

Versant Corporation

[email protected] 650-232-2431