using spring with nosql databases (springone china 2012)
DESCRIPTION
A look at how Spring Data simplifies the development of applications that use NOSQL databases such as Redis and MongoDB.TRANSCRIPT
Using Spring with NoSQL databasesChris Richardson, Author of POJOs in Action, Founder of the original CloudFoundry.com @crichardson [email protected] http://plainoldobjects.com/
Presentation goal
NoSQL databases: what, why and how
How Spring Data simplifies the development of NoSQL applications
About Chris
(About Chris)
About Chris()
About Chris
About Chris
http://www.theregister.co.uk/2009/08/19/springsource_cloud_foundry/
vmc push About-Chris
Developer Advocate
Signup at http://cloudfoundry.com
Agenda
• Why NoSQL?
• Overview of NoSQL databases
• Introduction to Spring Data
• Using Spring Data for Redis
• Using Spring Data for Mongo
• Deploying on Cloud Foundry
Relational databases are great...
• SQL
• High-level
• Sorting
• Aggregation
• ACID semantics
• Well supported
• JDBC
• Hibernate/JPA
• Spring
• Well understood
• Developers
• Operators
... but they have limitations
• Object/relational impedance mismatch
• Complicated to map rich domain model to relational schema
• Difficult to handle semi-structured data, e.g. varying attributes
• Schema changes
• Extremely difficult/impossible to scale
• Poor performance for some use cases
Solution: Spend Money
http://upload.wikimedia.org/wikipedia/commons/e/e5/Rising_Sun_Yacht.JPG
OR
http://www.trekbikes.com/us/en/bikes/road/race_performance/madone_5_series/madone_5_2/#
• Hire more DevOps• Use application-level sharding• Build your own middleware• …
Solution: Use NoSQL
Benefits• Higher performance• Higher scalability• Richer data-model• Schema-less
Drawbacks• Limited transactions• Relaxed consistency• Unconstrained data
Growing in popularity…
But don’t get too excited
Solution: Use NewSQL• Relational databases with SQL and ACID transactions
AND• New and improved architecture - designed for modern hardware• Radically better scalability and performance
• NewSQL vendors: Gemfire/SQLFire, VoltDB, ...
Future = multi-paradigm data storage for enterprise applications
IEEE Software Sept/October 2010 - Debasish Ghosh / Twitter @debasishg
Agenda
• Why NoSQL?
• Overview of NoSQL databases
• Introduction to Spring Data
• Using Spring Data for Redis
• Using Spring Data for Mongo
• Deploying on Cloud Foundry
• Advanced key-value store
• Written in C
• Very fast, e.g. 100K reqs/sec
• Optional persistence
• Transactions with optimistic locking
• Master-slave replication
• Sharding using client-side consistent hashing
Redis
K1 V1
K2 V2
... ...
Using Redis (via CLI)redis 127.0.0.1:6379> set foo 1OKredis 127.0.0.1:6379> get foo"1"
Datatypes:•Strings•Hashes•Maps•Lists•Sets•Sorted sets
redis 127.0.0.1:6379> sadd myset a(integer) 1redis 127.0.0.1:6379> sadd myset b(integer) 1redis 127.0.0.1:6379> smembers myset 1) "a"2) "b"redis 127.0.0.1:6379> srem myset a(integer) 1redis 127.0.0.1:6379> smembers myset1) "b"
Redis use cases• Replacement for Memcached
• Session state
• Cache of data retrieved from system of record (SOR)
• Replica of SOR for queries needing high-performance
• Handling tasks that overload an RDBMS
• Hit counts - INCR
• Most recent N items - LPUSH and LTRIM
• Randomly selecting an item – SRANDMEMBER
• Queuing – Lists with LPOP, RPUSH, ….
• High score tables – Sorted sets and ZINCRBY
• …
MongoDB
• Document-oriented database
• JSON-style documents: objects, lists, primitives
• Schema-less
• Transaction = update of a single document
• Rich query language for dynamic queries
• Geospatial queries
• Grid FS provides file storage
• Very fast, asynchronous writes
• Highly scalable and available
Server
Database: Food To Go
Collection: Restaurants
Data model = Binary JSON documents
{ "name" : "TGI Fridays", "type" : ”American", "serviceArea" : [ "94619", "94618" ], "openingHours" : [ { "dayOfWeek" : "Wednesday", "open" : 1730, "close" : 2230 } ], "_id" : ObjectId("4bddc2f49d1505567c6220a0")}
Sequence of bytes on disk è fast i/o
MongoDB CLI> r = {name: 'Ajanta'}> db.restaurants.save(r)> r{ "_id" : ObjectId("4e555dd9646e338dca11710c"), "name" : "Ajanta" }> r = db.restaurants.findOne({name:"Ajanta"}){ "_id" : ObjectId("4e555dd9646e338dca11710c"), "name" : "Ajanta" }> r.type= "Indian”> db.restaurants.save(r)> db.restaurants.update({name:"Ajanta"}, {$set: {name:"Ajanta Restaurant"}, $push: { menuItems: {name: "Chicken Vindaloo"}}})> db.restaurants.find(){ "_id" : ObjectId("4e555dd9646e338dca11710c"), "menuItems" : [ { "name" :
"Chicken Vindaloo" } ], "name" : "Ajanta Restaurant", "type" : "Indian" }> db.restaurants.remove(r.id)
MongoDB query by example{ serviceArea:"94619", openingHours: { $elemMatch : { "dayOfWeek" : "Monday", "open": {$lte: 1800}, "close": {$gte: 1800} } }}
DBCursor cursor = collection.find(qbeObject); while (cursor.hasNext()) { DBObject o = cursor.next(); … }
Find a restaurant that serves the 94619 zip code and is open at 6pm on a Monday
Mongos
Scaling MongoDBShard 2
Mongod(replica)
Mongod(master)
Mongod(replica)
Shard 1Mongod(replica)
Mongod(master)
Mongod(replica)
Mongos
Client
Config Server
mongod
mongod
mongod
MongoDB use cases
• Use cases
• High volume writes
• Complex data
• Semi-structured data
• Who is using it?
• Shutterfly, Foursquare
• Bit.ly Intuit
• SourceForge, NY Times
• GILT Groupe, Evite,
• SugarCRM
Other NoSQL databases
http://nosql-database.org/ lists 122+ NoSQL databases
Type Examples
Extensible columns/Column-oriented HbaseSimpleDB, DynamoDBCassandra
Graph Neo4j
Key-value Voldemort, Riak
Document CouchDbSorry if
I left
out you
r favori
te
Agenda
• Why NoSQL?
• Overview of NoSQL databases
• Introduction to Spring Data
• Using Spring Data for Redis
• Using Spring Data for Mongo
• Deploying on Cloud Foundry
Spring Data is here to help
http://www.springsource.org/spring-data
NoSQL databases
For
Spring Data sub-projects
• Relational
• JPA
• JDBC Extensions
• NoSQL
• Redis
• Mongo
• HBase
• Neo4j
• Gemfire
• Lucene
• QueryDSL
• Big Data
• Hadoop
• HDFS and M/R
• Hive
• Pig
• Cascading
• Splunk
• Access
• REST
What you get
• Template classes that hide the boilerplate code
• Auto-generated (generic) repositories for some NOSQL databases
• Java ⇔ NoSQL mapping
• Cross Store Persistence
• Support in Roo
Get the book!
Agenda
• Why NoSQL?
• Overview of NoSQL databases
• Introduction to Spring Data
• Using Spring Data for Redis
• Using Spring Data for Mongo
• Deploying on Cloud Foundry
Redis challenges
• Connection management: need to get and reliably close connections
• Data mapping: application objects ⇔ Redis binary/strings
• Multiple client libraries with gratuitously different APIs
Spring Data for Redis
• Low-level - RedisConnection(Factory)
• Supports Jedis, Jredis, Rjc and Srp
• Insulates client code from underlying library
• High-level - RedisTemplate
• Builds on RedisConnection(Factory)
• Connection management
• Pluggable Java ⇔ binary conversion
• Support classes:
• Collections-backed by RedisTemplate
• Atomic Counters
• Support for Redis pub/sub
Low-level API = RedisConnection(Factory)
Using RedisConnectionFactorypublic class LowLevelRedisTest {
@Autowired private RedisConnectionFactory redisConnectionFactory;
@Test public void testLowLevel() { RedisConnection con = null; try { con = redisConnectionFactory.getConnection();
byte[] key = "foo".getBytes(); byte[] value = "bar".getBytes(); con.set(key, value);
byte[] retrievedValue = con.get(key);
Assert.assertArrayEquals(value, retrievedValue);
} finally { if (con != null) { con.close(); } } }
Ugly byte arrays L
Library independent code J
Need to clean up L
Configuring RedisConnectionFactory
@Configurationpublic class RedisConfiguration {
@Value("${databaseHostName}") protected String databaseHostName;
@Bean public RedisConnectionFactory jedisConnectionFactory() { JedisConnectionFactory factory = new JedisConnectionFactory(); factory.setHostName(databaseHostName); factory.setPort(6379); factory.setUsePool(true); return factory; }
}
High-level API = RedisTemplate
• Builds on RedisConnection(Factory)
• Analogous to JdbcTemplate
• Parameterized type
• K - Key type
• V – Value type
• Handles Java Key/Value ⇔ Redis byte[]
• Maps Redis exceptions ⇒
DataAccessException
• StringRedisTemplate
• Extends RedisTemplate<String, String>
• Keys and values are Strings
Using StringRedisTemplate
public class RedisTemplateTest {
@Autowired private StringRedisTemplate stringRedisTemplate;
@Test public void testGetAndSet() { stringRedisTemplate.opsForValue().set("foo", "bar"); assertEquals("bar", stringRedisTemplate.opsForValue().get("foo")); } @Test public void testHashOps() { stringRedisTemplate.opsForHash().put("myHash", "myKey", "value");
assertEquals("value", stringRedisTemplate.opsForHash().get("myHash", "myKey"));
assertEquals(Collections.singleton("myKey"), stringRedisTemplate.opsForHash().keys("myHash"));
assertEquals(Collections.singletonMap("myKey", "value"), stringRedisTemplate.opsForHash().entries("myHash")); }
Converts between Strings and byte[]
Returns KV type specific interface
Configuring StringRedisTemplate
@Configurationpublic class RedisConfiguration {
@Bean public RedisConnectionFactory jedisConnectionFactory() { … }
@Bean public StringRedisTemplate stringRedisTemplate(RedisConnectionFactory factory) { StringRedisTemplate template = new StringRedisTemplate(); template.setConnectionFactory(factory); return template; }}
RedisTemplate: Java objects ⇔ binary data
• DefaultSerializer - defaults to JdkSerializationRedisSerializer
• KeySerializer
• ValueSerializer
• HashKeySerializer
• HashValueSerializer
StringRedisTemplate uses StringRedisSerializer
• Stores keys and values as Strings
Register serializers to override the default behavior
Converted to JSON by RedisTemplate
Redis-backed Collections
@Test public void testRedisSet() {
Set<String> mySet = new DefaultRedisSet<String>("mySet", stringRedisTemplate);
Assert.assertTrue(mySet.isEmpty());
mySet.add("a");
assertEquals(Collections.singleton("a"), stringRedisTemplate.opsForSet().members("mySet"));
}
The key
Redis Atomic Counters
Redis Pub/Sub - Consumer
@Configurationpublic class RedisConfiguration {
@Bean public MyRedisSubscriber myListener() { return new MyRedisSubscriber(); } @Bean public RedisMessageListenerContainer redisMessageListenerContainer( RedisConnectionFactory redisConnectionFactory, MyRedisSubscriber myListener) { RedisMessageListenerContainer c = new RedisMessageListenerContainer();
c.setConnectionFactory(redisConnectionFactory); c.addMessageListener(new MessageListenerAdapter(myListener), new ChannelTopic("myChannel")); return c; }
public class MyRedisSubscriber { String message;
void handleMessage(String text) { this.message = text; } }
Redis Pub/Sub - Producer
public class RedisPubSubTest {
@Autowired private StringRedisTemplate stringRedisTemplate;
@Autowired private MyRedisSubscriber myListener; @Test public void testPubSub() throws InterruptedException { … stringRedisTemplate.convertAndSend("myChannel", "hello"); TimeUnit.SECONDS.sleep(4); Assert.assertEquals("hello", myListener.message); }}
Redis caching support
Template needs to (de)serialize K and VKVs = <prefix + K, V>
Sorted set of all keys for clear()
Agenda
• Why NoSQL?
• Overview of NoSQL databases
• Introduction to Spring Data
• Using Spring Data for Redis
• Using Spring Data for Mongo
• Deploying on Cloud Foundry
MongoDB API usage patterns
• Create and store Mongo singleton
• Externalized server host, port etc.
• Inserts/Updates
• Map application POJO ⇒ DBObject
• mongo.getDatabase(…).getCollection(…)
• Partial document updates
• Configure asynchronous vs. synchronous writes
• Queries
• Construct query object
• mongo.getDatabase(…).getCollection(…)
• Iterate through Cursor
• Map DBObject ⇒ application POJO
⇒ Higher-level than JDBC but still
repetitive, …
Spring Data - MongoDB
• MongoTemplate
• Query, Criteria, and Update DSLs
• Generic repositories
• Querydsl integration
• Cross-store persistence
• GeoSpatial integration
• Map-Reduce integration
• GridFS support
MongoTemplatedatabaseNameuserIdPassworddefaultCollectionName
writeConcernwriteResultChecking
save()insert()remove()updateFirst()findOne()find()…
MongoTemplate
Mongo(Java Driver class)
<<interface>>MongoConvertor
write(Object, DBObject)read(Class, DBObject)
uses
POJO ó DBObjectmappingSimplifies data access
Translates exceptions
MongoMappingConverter
Example entitypublic class Restaurant { private String id; private String name; private List<MenuItem> menuItems;
public Restaurant(String name) { this.name = name; … }
...}
public class MenuItem { private String name; private double price;
public MenuItem(String name, double price) { this.name = name; this.price = price; }
.....
Spring Data uses fields and non-default constructors
Example data access code
@Repositorypublic class RestaurantRepository {
@Autowired private MongoTemplate mongoTemplate; public void add(Restaurant restaurant) { mongoTemplate.save(restaurant); }
public List<Restaurant> findRestaurantsByName(String restaurantName) { return mongoTemplate.find( query(where("name").is(restaurantName)), Restaurant.class); }
A document in the restaurant collection
{ "_id" : ObjectId("4d977f55d3fe3119c904e026"), "_class" : "net.chrisrichardson.mongodb.example.mongotemplate.Restaurant", "name" : "Ajanta" "menuItems" : [ { "name" : "Tandoori Portobello Mushrooms", "price" : 5.5 }, { "name" : "Duck Curry Kerala", "price" : 15 } ]}
Spring MongoDB Example - Config 1@Configurationpublic class MongoExampleConfig extends AbstractMongoConfiguration {
private @Value("#{mongoDbProperties.databaseName}") String mongoDbDatabase;
private @Value("#{mongoDbProperties.host}") String mongoDbHost;
public Mongo mongo() throws Exception { return new Mongo(mongoDbHost); }
@Override protected String getDatabaseName() { return mongoDbDatabase; }...}
databaseName=demo1host=192.168.253.150
mongodb.properties:
<beans> <context:annotation-config/> <context:component-scan base-package="net.chrisrichardson.mongodb.example"/>
<util:properties id="mongoDbProperties" location="mongodb.properties"/>
</beans>
External Config
Defines a MongoTemplate
Spring MongoDB Example - Config 2<bean id="mongoTemplate" class="org.springframework.data.mongodb.core.MongoTemplate"> <constructor-arg ref="mongoFactory"/></bean>
<mongo:db-factory id="mongoFactory" host= "#{mongoDbProperties.host}"dbname="#{mongoDbProperties.databaseName}" />
<util:properties
id="mongoDbProperties" location="mongodb.properties"/>
Update example@Repositorypublic class RestaurantRepository { public void addMenuItem(String restaurantId, MenuItem newMenuItem) { mongoTemplate.updateFirst( query(where("_id").is(new ObjectId(restaurantId))), new Update().push("menuItems", newMenuItem), Restaurant.class); }
Atomic, in-place update of document
Geospatial example 1
@Componentclass MongoFriendService extends FriendService {
@Autowired var mongoTemplate: MongoTemplate = _
@PostConstruct def createGeoIndex { val dbo = new BasicDBObject dbo.put("location", "2d") mongoTemplate.getCollection("friendRecord").ensureIndex(dbo) }
Create geospatial 2d index
Collection name
case class FriendRecord(id : String, name : String, location : Point)
Geospatial example 2 - finding nearby
@Componentclass MongoFriendService extends FriendService {
override def findNearbyFriends(request: NearbyFriendsRequest) = { val location = new Point(request.longitude, request.latitude) val distance = new Distance(3, Metrics.MILES) val query = NearQuery.near(location).maxDistance(distance) val result = mongoTemplate.geoNear(query, classOf[FriendRecord])
val nearby = result.getContent.map(_.getContent) FindNearbyFriendsResponse(nearby.map(f => FriendInfo(f.name, f.id))) }
Callbacks – access driver API with exception translation
@Test public void testDbCallback() { Restaurant ajanta = makeAjantaRestaurant(); restaurantRepository.add(ajanta); assertCollectionExists("restaurants2"); }
private Void assertCollectionExists(final String collectionName) { return mongoTemplate.execute(new DbCallback<Void>(){ @Override public Void doInDB(DB db) { Set<String> collectionNames = db.getCollectionNames(); Assert.assertTrue("Missing from " + collectionNames, collectionNames.contains(collectionName)); return null; }}); }
Exceptions are translated
Defining a Mongo Generic Repository
interface PersonRepository extends MongoRepository<Person, ObjectId> { List<Person> findByLastname(String lastName);}
public class Person { private ObjectId id; private String firstname; private String lastname;… getters and setters}
Person p = new Person("John", "Doe");personRepository.save(p);
Person p2 = personRepository.findOne(p.getId());
List<Person> johnDoes = personRepository.findByLastname("Doe"); assertEquals(1, johnDoes.size());
Mongo Repository configuration<bean>
<mongo:repositories base-package="net.chrisrichardson.mongodb.example.mongorepository" mongo-template-ref="mongoTemplate" />
</beans>
Scans classpath looking for subtypes of MongoRepository in the base package
Richer mapping@Document(collection=”people”)public class Person {
@Id private ObjectId id; private String firstname; @Indexed private String lastname;
@PersistenceConstructor public Person(String firstname, String lastname) { this.firstname = firstname; this.lastname = lastname; }….}
Annotations define mapping: @Document, @Id, @Indexed, @PersistanceConstructor, @CompoundIndex, @DBRef, @GeoSpatialIndexed, @Value
Map fields instead of properties ⇒ no getters or setters
required
Non-default constructorIndex generation
Richer mapping configuration@Configurationpublic class MongoExampleConfig extends AbstractMongoConfiguration { private @Value("#{mongoDbProperties.databaseName}") String mongoDbDatabase;
private @Value("#{mongoDbProperties.host}") String mongoDbHost;
@Override public Mongo mongo() throws Exception { return new Mongo(mongoDbHost); } @Override public String getDatabaseName() { return mongoDbDatabase; }
@Override public String getMappingBasePackage() { return Person.class.getPackage().getName(); }}
Defines MongoTemplate bean
Configures classpath scanning
Support for the QueryDSL project
QPerson person = QPerson.person;
Predicate predicate = person.homeAddress.street1.eq("1 High Street") .and(person.firstname.eq("John"))
List<Person> people = personRepository.findAll(predicate);
assertEquals(1, people.size());assertPersonEquals(p, people.get(0));
Generated from domain model class
Type-safe composable queries
Cross-store/polyglot persistence
@Entitypublic class Person { // In Database @Id private Long id; private String firstname; private String lastname; // In MongoDB @RelatedDocument private Address address;
{ "_id" : ObjectId(”….."), "_entity_id" : NumberLong(1), "_entity_class" : "net.. Person", "_entity_field_name" : "address", "zip" : "94611", "street1" : "1 High Street", …}
Person person = new Person(…);entityManager.persist(person);
Person p2 = entityManager.find(…)
Agenda
• Why NoSQL?
• Overview of NoSQL databases
• Introduction to Spring Data
• Using Spring Data for Redis
• Using Spring Data for Mongo
• Deploying on Cloud Foundry
Using Mongo and Redis with Cloud Foundry
• Create a Mongo or Redis service
• Bind the service to your application
• Access the service
• Via auto-reconfiguration
• Using <cloud:*/> namespace
Creating a Redis Server
Deploying a Redis application
Redis bean definitions
USING THE APPLICATION
About <cloud:redis-connection-factory/>
<cloud:redis-connection-factory id="redisConnectionFactory" service-name="redis1" />
Use when multiple services are bound
Deploying a Mongo application
MongoDB bean definitions
Using the Mongo Application
About <cloud:mongo-db-factory/><cloud:mongo-db-factory id="mongoFactory" service-name="mongo1" > <cloud:mongo-options connections-per-host="..." max-wait-time="..." /></cloud:mongo-db-factory>
Use when multiple services are bound
]
NoSQL and Caldecott
• Caldecott let’s you tunnel to a NoSQL service
• Use Redis CLI
• Explore database, adhoc operations
• ...
• Use Mongo CLI etc
• Explore database, adhoc operations
• Mongo dump/restore
• ...
Summary
• NoSQL databases sometimes offer a combination of:
• Higher scalability and performance
• Schema less, richer data models
• Spring Data simplifies the development of NoSQL applications
• Cloud Foundry supports Mongo and Redis
84
Cloud Foundry 启动营在www.cloudfoundry.com注册账号并成功上传应用程序,即可于12月8日中午后凭账号ID和应用URL到签到处换取Cloud Foundry主题卫衣一件。
85
iPhone5 等你拿第二天大会结束前,请不要提前离开,将填写完整的意见反馈表投到签到处的抽奖箱内,即可参与“iPhone5”抽奖活动。
86
Birds of a Feather 专家面对面所有讲师都会在课程结束后,到紫兰厅与来宾讨论课程上的问题