MongoDB@BayernMigration from RDBMS, Problems, Unit-Testing

Christian Brensing

Bayerisches Landesamt für Statistik und Datenverarbeitung

IuK / Rechenzentrum Süd

• Template-Processor (~ JSP)

• ODF and RTF

• PDF / PostScript postprocessing

• Groovy, Python, Ruby, Tcl

• In Production using RDBMS since 2008

• Central HR system (SAP) uses BayText to create output

Client for developing the templates (BayText-IDE)

ODF-Template using Ruby

Generated document

2 MillionGenerated documents per year

Tiny dataset50,000 records and 10 GB BLOB

a small system after all

So why migrate?

Because we can

developers.pop()Smaller teams

Easy to learnRead one book and you're done

Simple mapping

nodes

groups

templates

document_usages

folder_configurations

codes

permissions

requests

documents

folders

structures template_tags

roles

roles_roles

bundles

componentsdynafields

scripts

nodes

roles

properties

GridFS

{! "_id" : ObjectId(),! "_type" : "Document",! "updated_at" : ISODate("2007-12-06T10:04:00.543Z"),! "updated_by" : "maggie",! "version" : 1,! "path" : "foo.bar.Document",! "description" : "",! "language" : "ruby",! "format" : "odf",! "autoload_uplevel" : -1,! "requests" : [! {! "name" : "Test",! "description" : "foo", ! "xml" : zlib("<?xml ...?>")! }, ! ... ! ],! "structure" : [! { ! "alias" : "some alias",! "component_path" : "foo.bar.Document$Document",! "parent", -1 ! },! ... ! ], ! } ! "tags" : ["foo", "bar", "baz"]!}

public class RoleReadConverter implements Converter<DBObject, Role> {! @Override! public Role convert(DBObject source) {! Role role = new Role();! role.setName((String) source.get("name"));! ...! return role;! }!}!!!public class RoleWriteConverter implements Converter<Role, DBObject> {! @Override! public DBObject convert(Role source) {! return new BasicDBObject()! .append("name", role.getName())! .append(...);! }!}!!!// Convert a DBObject to an Entity!conversionService.convert(collection.findOne(...));

Easy mapping using Spring-ConversionService

Performanceup to 10x

ORM no moreThe MongoDB driver is all you need

Obstacles?

Mentality

Referential IntegrityMulti-Document-Update on denormalized schema

{! "_id" : ObjectId(),! "_type" : "Folder", ! "updated_at" : ISODate("2007-12-06T10:04:00.543Z"),! "updated_by" : "maggie",! "version" : 1,! "path" : "foo.bar.Folder",! "description" : "bla",! "bundles" : [! ... ! ]! "document_usages" : [! {! "name" : "foo",! "description" : "bla", ! "exec_order" : 5, ! "print_copies" : 2,! "type": "FAIR_COPY",! "document_path" : "a.b.Document",! "bundle" : "bar"! },! ...!}

Linking documents via a path instead of ObjectId

But paths are mutable!All references must be updated

Approach

• Pseudo-Transaction with an update locking mechanism in a custom oplog collection

• Periodical repair jobs finishing failed operations • Extended interpretation of eventual consistency

Unit-TestsContinuous Integration

F.I.R.S.T.

IsolatedTests are using a unique DB per JVM

@Configuration!@Profile("test")!public class TestSpringConfiguration extends SpringConfiguration {! @Override! public void init() {! setTestProperties();! ! super.init();! ! // Delete test db! db().dropDatabase();! }! ! private void setTestProperties() {! String dbname = env.getProperty("db.name", "baytext_test_" + getSimpleUsername());! String host = env.getProperty("db.host", "mongodb-dev.db.rz-sued.bybn.de");! String port = env.getProperty("db.port", "27016");! ! cmProperties.setProperty("servers", String.format("%s:%s", host, port));! cmProperties.setProperty("name", dbname);! }!! // OS-Username without prefix (e.g. maggie instead of lfstad-maggie)! private static String getSimpleUsername() {! String username = SystemUtils.USER_NAME;! int indexOfDash = username.indexOf('-');! return indexOfDash != -1 ? username.substring(indexOfDash + 1) : username;! }!}

Test-ApplicationContext

RepeatableCollections are dropped before each test method

public class MongoTestExcecutionListener extends AbstractTestExecutionListener {! @Override! public void beforeTestMethod(TestContext testContext) throws Exception {! purgeCollections();! }! ! private void purgeCollections() {! DB db = MongoDBHolder.getDB();! for (String collectionName : db.getCollectionNames()) {! if (collectionName.startsWith("fs.") || collectionName.startsWith("system.")) {! continue;! }! DBCollection collection = db.getCollection(collectionName);! if (!collection.isCapped()) {! collection.drop();! }! }! }!}

Spring-TestExecutionListener

@RunWith(SpringJUnit4ClassRunner.class)!@ActiveProfiles({"test"})!@ContextConfiguration(classes = TestSpringConfiguration.class)!@TestExecutionListeners({! DependencyInjectionTestExecutionListener.class,! DirtiesContextTestExecutionListener.class,! MongoTestExcecutionListener.class!})!public abstract class MongoTestSupport {!}!!!!public class DocumentRepositoryTest extends MongoTestSupport {!}!

Test base class

95% Coverage

Summary

• Developing the data layer is fun again • Reduced complexity • Flat learning curve compared to SQL/ORM • use_mongo() unless transactions_required

Thank you!