webinar: strongly typed languages and flexible schemas
TRANSCRIPT
Strongly Typed Languages and Flexible Schemas
3
Agenda
Strongly Typed Languages
Flexible Schema Databases
Change Management
Strategies
Tradeoffs
Strongly Typed Languages
"A programming language that requires a variable to be defined as well as the variable it is"
Flexible Schema Databases
7
Traditional RDMS
create table users (id int, firstname text, lastname text);
Table definition
Column structure
8
Traditional RDMS
Table with checks
create table cat_pictures(
id int not null,
size int not null,
picture blob not null,
user_id int,
primary key (id),
foreign key (user_id) references users(id));
Null checks
Foreign and Primary key checks
9
Traditional RDMS
users cat_pictures
1 N
10
Is this Flexible?
• What happens when we need to change the schema?– Add new fields– Add new relations– Change data types
• What happens when we need to scale out our data structure?
11
Flexible Schema Database
Document Graph Key Value
12
Flexible Schema
• No mandatory schema definition• No structure restrictions• No schema validation process
13
We start from code
public class CatPicture {
int size;byte[] blob;
}
public class User {
int id;String firstname;String lastname;
CatPicture[] cat_pictures;
}
14
Document Structure
{ _id: 1234, firstname: 'Juan', lastname: 'Olivo', cat_pictures: [ { size: 10, picture: BinData("0x133334299399299432"), } ]}
Rich Data Types
Embedded Documents
15
Flexible Schema Databases
• Challenges–Different Versions of Documents–Different Structures of Documents–Different Value Types for Fields in
Documents
16
Different Versions of Documents
Same document across time suffers changes on how it represents data
{ "_id" : 174, "firstname": "Juan" }
{ "_id" : 174, "firstname": "Juan", "lastname": "Olivo" }
First Version
Second Version
{ "_id" : 174, "firstname": "Juan", "lastname": "Olivo" , "cat_pictures": [{"size": 10, picture: BinData("0x133334299399299432")}]}
Third Version
17
Different Versions of Documents
Same document across time suffers changes on how it represents data
{ "_id" : 174, "firstname": "Juan" }
{ "_id" : 174, "name": { "first": "Juan", "last": "Olivo"} }
Different Structure
18
Different Structures of Documents
Different documents coexisting on the same collection
{ "_id" : 175, "brand": "Ford", "model": "Mustang", "date": ISODate("XXX") }
{ "_id" : 174, "firstname": "Juan", "lastname": "Olivo" }
Within same collection
19
Different Data Types for Fields
Different documents coexisting on the same collection
{ "_id" : 174, "firstname": "Juan", "lastname": "Olivo", "bdate": 1224234312}
{ "_id" : 175, "firstname": "Paco", "lastname": "Hernan", "bdate": "2015-06-27"}
{ "_id" : 176, "firstname": "Tomas", "lastname": "Marce", "bdate": ISODate("2015-06-27")}
Same field, different data type
Change Management
21
Change Management
Versioning Class Loading
How to set correct data format versioning?
What mechanisms are out there to make this work ?
Strategies
23
Strategies
• Decoupling Architectures• ODM'S• Versioning• Data Migrations
Decoupled Architectures
25
Strongly Coupled
26
Becomes a mess in your hair…
Coupled Architectures
DatabaseApplication A
Application C
Application B Let me perform some schema
changes!
Decoupled Architecture
DatabaseApplication A API
Application C
Application B
29
Decoupled Architectures
• Allows the business logic to evolve independently of the data layer
• Decouples the underlying storage / persistency option from the business service
• Changes are "requested" and not imposed across all applications
• Better versioning control of each request and it's mapping
ODM's
31
ODM
• Reduce impedance between code and Databases• Data management facilitator • Hides complexity of operators• Tries to decouple business complexity with "magic"
recipes
32
Spring Data
• POJO centric model• MongoTemplate || CrudRepository
extensions to make the connection to the repositories
• Uses annotations to override default field names and even data types (data type mapping)
public interface UserRepository extends MongoRepository<User, Integer>{
}
public class User {
@Idint id;
@Field("first_name")String firstname;String lastname;
33
Spring Data Document Structure
{ "_id": 1, "first_name": "first", "lastname": "last", "catpictures": [ { "size": 10, "blob": BinData(0, "Kr3AqmvV1R9TJQ==") }, ]}
34
Spring Data Considerations
• Data formats, versions and types still need to be managed
• Does not solve issues like type validation out-of-box• Can make things more complicated but more
"controllable"@Field("first_name")String firstname;
35
Morphia
• Data source centric• Will do all the discovery of POJO's for
given package• Also uses annotations to perform
overrides and deal with object mapping
@Entity("users")public class User {
@Idint id;String firstname;String lastname;
morphia.mapPackage("examples.odms.morphia.pojos");
Datastore datastore = morphia.createDatastore(new MongoClient(), "morphia_example");datastore.save(user);
36
Morphia Document Structure
{ "_id": 1, "className": "examples.odms.morphia.pojos.User", "firstname": "first", "lastname": "last", "catpictures": [ { "size": 10, "blob": BinData(0, "Kr3AqmvV1R9TJQ==") }, ]}
Class Definition
37
Morphia Considerations
• Enables better control at Class loading• Also facilitates, like Spring Data, the field overriding (tags
to define field keys)• Better support for Object Polymorphism
Versioning
39
Versioning
Versioning of data structures (specially documents) can be very helpful
Recreate documents over time
Flow Control
Data / Field Multiversion Requirements
Archiving and History Purposes
40
Versioning – Option 0
Change existing document each time there is a write with monotonically increasing version number inside
{ "_id" : 174, "v" : 1, "firstname": "Juan" }
{ "_id" : 174, "v" : 2, "firstname": "Juan", "lastname": "Olivo" }
{ "_id" : 174, "v" : 3, "firstname": "Juan", "lastname": "Olivo", "gender": "M" }
> db.users.update( {"_id":174 } , { {"$set" :{ ... }, {"$inc": { "v": 1 }} } )
Increment field value
41
Versioning – Option 1
Store full document each time there is a write with monotonically increasing version number inside
{ "docId" : 174, "v" : 1, "firstname": "Juan" }
{ "docId" : 174, "v" : 2, "firstname": "Juan", "lastname": "Olivo" }
{ "docId" : 174, "v" : 3, "firstname": "Juan", "lastname": "Olivo", "gender": "M" }
> db.users.insert( {"docId":174 …})
> db.docs.find({"docId":174}).sort({"v":-1}).limit(-1);
Find always latest version
42
Versioning – Option 2
Store all document versions inside a single document.
> db.users.update( {"_id": 174 } , { {"$set" :{ "current": ... }, {"$inc": { "current.v": 1 }}, {"$addToSet": {"prev": {... }}} } )
Current value
{ "_id" : 174, "current" : { "v" :3, "attr1": 184, "attr2" : "A-1" }, "prev" : [ { "v" : 1, "attr1": 165 }, { "v" : 2, "attr1": 165, "attr2": "A-1" } ]}
Previous values
43
Versioning – Option 3
Keep collection for "current" version and past versions
> db.users.find( {"_id": 174 })
> db.users_past.find( {"pid": 174 })
{ "pid" : 174, "v" : 1, "firstname": "Juan" }
{ "pid" : 174, "v" : 2, "firstname": "Juan", "lastname": "Olivo" }
{ "_id" : 174, "v" : 3, "firstname": "Juan", "lastname": "Olivo", "gender": "M" }
Previous versions collection
Current collection
44
Versioning
Schema Fetch 1 Fetch Many Update Recover if Fail
0) Increment Version
Easy, Fast Fast Easy Medium N/A
1) New Document
Easy, Fast Not Easy, Slow
Medium Hard
2) Embedded in Single Doc
Easy, Fastest
Easy, Fastest Medium N/A
3) Separate Collection
Easy, Fastest
Easy, Fastest Medium Medium, Hard
Migrations
46
Migrations
Several types of "Migrations":
Add/Remove Fields
Change Field Names
Change Field Data Type
Extract Embedded Document into Collection
47
Add / Remove Fields
For Flexible Schema Database this is our Bread & Butter
{ "_id" : 174, "firstname": "Juan", "lastname": "Olivo", "gender": "M" }
{ "_id" : 174, "firstname": "Juan", "lastname": "Olivo", "newfield": "value" }
> db.users.update( {"_id": 174}, {"$set": { "newfield": "value" }, "$unset": {"gender":""} })
48
Change Field Names
Again, programmatically you can do it
{ "_id" : 174, "firstname": "Juan", "lastname": "Olivo",}
{ "_id" : 174, "first": "Juan", "last": "Olivo" }
> db.users.update( {"_id": 174}, {"$rename": { "firstname": "first", "lastname":"last"} })
49
Change Field Data Type
Align to a new code change and move from Int to String
{..."bdate": 1435394461522} {..."bdate": "2015-06-27"}
1) Batch Process
2) Aggregation Framework
3) Change based on usage
50
Change Field Data Type1) Batch Process – bulk api
public void migrateBulk(){DateFormat df = new SimpleDateFormat("yyyy-MM-DD");...List<UpdateOneModel<Document>> toUpdate =
new ArrayList<UpdateOneModel<Document>>();for (Document doc : coll.find()){
String dateAsString = df.format( new Date( doc.getInteger("bdate", 0) ));Document filter = new Document("_id", doc.getInteger("_id"));Document value = new Document("bdate", dateAsString);Document update = new Document("$set", value);
toUpdate.add(new UpdateOneModel<Document>(filter, update));}coll.bulkWrite(toUpdate);
51
Change Field Data Type1) Batch Process – bulk api
public void migrateBulk(){...for (Document doc : coll.find()){
...}coll.bulkWrite(toUpdate);
Is there any problem with this?
52
Change Field Data Type1) Batch Process – bulk api
public void migrateBulk(){...//bson type 16 represents int32 data typeDocument query = new Document("bdate", new Document("$type", "16"));for (Document doc : coll.find(query)){
...}
coll.bulkWrite(toUpdate);More efficient filtering!
53
Extract Document into CollectionNormalize your schema
{"size": 10, picture: BinData("0x133334299399299432")}{ "_id" : 174, "firstname": "Juan", "lastname": "Olivo",}
> db.users.aggregate( [ {$unwind: "$cat_pictures"}, {$project: { "_id":0, "uid":"$_id", "size": "$cat_pictures.size", "picture": "$cat_pictures.picture"}}, {$out:"cats"}])
{ "_id" : 174, "firstname": "Juan", "lastname": "Olivo" , "cat_pictures": [{"size": 10, picture: BinData(0, "m/lhLlLmoNiUKQ==")}]}
{"size": 10, "picture": BinData(0, "m/lhLlLmoNiUKQ==")}
Tradeoffs
55
Tradeoffs
Positives Penalties
Decoupled Architecture - Should be your default approach
- Clean Solution - Scalable
N/A
Data Structures Variability - Reflects Nowadays data structures
- You can push decisions for later
- More complex code base
Data Structures Strictness - Simple to maintain- Always aligned with your
code base
- Will eventually need Migrations
- Restricts your code iterations
Recap
57
Recap
• Flexible and Dynamic Schemas are a great tool– Use them wisely – Make sure you understand the tradeoffs– Make sure you understand the different strategies and
options
• Works well with Strongly Typed Languages
58
Free Educationhttps://university.mongodb.com/courses/M101J/about
Obrigado!• Norberto Leite• Technical Evangelist• http://www.mongodb.com/norberto• [email protected]• @nleite
