building a distributed reservation system using cassandra

Jeff Carpenter and Andrew BakerChoice Hotels International

Building a Distributed Reservation System Using Cassandra

1 The Problem – Replacing a Reservation System

2 Managing Consistency

3 Microservices and Data Integrity

4 Schema Evolution

5 Data Retention and TTL

6 Performance and Cost Tradeoffs2© DataStax, All Rights Reserved.

© DataStax, All Rights Reserved. 3

Central Reservation System Interfaces

CRSProperty Systems

Web and Mobile

External Channels Reportin

g & Billing

Customer &

Loyalty


Current Reservation System – By The Numbers

25 years6,000 hotels

50transactions / second4,000distribution channels

1 instance


Architecture Tenets

MicroservicesCloud-nativeRules-based

Open Source InfrastructureStable, Scalable, Secure


Non-relational

Rules-based

Reporting & Analytics

Cloud deployment

RESTful services

High availability


Our configuration

• 18 nodes• Cassandra 2.2.X • I2.2XL (Smaller in Dev/Test)• 1 TB and growing

• 3 regions• AWS VPC• Direct Connect• Legacy systems in on-

prem data center

C*


Project Timeline

Inception• Proof of

concept

Beta• Initial

Capability• Beta Release• <1%

production traffic

Release 1• Full

Capability• ~10%

production traffic

Completion• 100%

production traffic

• Legacy System Retirement

Look, Ma, 100K writes/sec!

Why are my repairs failing?

We got this!


Key Data Types

rates inventoryhotels reservations


Key Data Types → Microservices

Hotel Service

Booking Service

Rates Service

Shopping Service

Data Maintenance Apps

Inventory Service

Reservation Service

Inventory keyspace

Rates keyspace

Hotels keyspace

Reservations keyspace


Varying Consistency Needs

Eventual consistency Immediate consistency

ratesreservationshotels inventory


Distributed Transactions, Anyone?

Commit thecontract

Reservethe inventory

Booking Service

Data Maintenance Apps

Inventory Service

Reservation Service

inventory

reservations

Data synchronization

Alternatives to Distributed Transactions

Approach Example Scope

Lightweight Transaction Updating inventory counts Data Tier

Logged Batch Writing to multiple denormalized hotel tables Data Tier

Retrying failed calls Data synchronization, reservation processing Service

Compensating processes Verifying reservation processing System


Eventual consistency

Strong consistency


Where is my data?

Create Block

C*

node

node

node

nodeUpdate

Inventory

Check Block & Adjust Inventory

Count

LOCAL_ONE

LOCAL_QUORUM

LOCAL_QUORUM


Configurable Consistency Levels

C*

node

node

node

nodeA LOCAL_ONE

C*

node

node

node

nodeB LOCAL_QUORUM

Test

© DataStax, All Rights Reserved. 16© DataStax, All Rights Reserved. 16

Cross Region Issues


ServiceReliability

ClusterBuilder. addContactPoint()

ClusterBuilder. addContactPoints()

VS

Java Driver

C*

node

node

node

node X


Complex Time Queries

Shoppingrequest

Rate data

I can’t do a range query on departure date

If I do a range query on arrival date…

Time


Keyspace edge_hotel

Denormalization Gone Wild (aka “Hotel Access Patterns”)

Locate hotel by identifier

Find hotels within X miles

of point Y

Find hotels by city, state, country

Find hotels by postal

code

Hotels by amenity

Find hotels by brand

hotels_by_id

hotels_by_brand

hotels_by_postal_code

…

Hotels by this

Hotels by that

Hotels by something

else

20

Schema Evolution

© DataStax, All Rights Reserved.

CREATE TABLE rates_by_hotel (id text, hotel_id text, code text, name text, product_ids set<text>, categories set<text>, PRIMARY KEY ((hotel_id), code));

ALTER TABLE rates_by_hotelDROP code;

ALTER TABLE rates_by_hotelADD code int;

schema.cql

001_code_to_integer.cql

001_rollback.cql

ALTER TABLE rates_by_hotelDROP code;

ALTER TABLE rates_by_hoteladd code text;

Size and Cost Estimation


47%

37%

7%

9%

Data SizeInventory RatesHotels Reservations

Schemas

Sizes

Chebotko Formulas

Estimates

• String length• Collection size• Partition and

row counts


TTL for Data Cleanup

Now

Time

Yesterday’s data is ancient history

Rate + Inventory Data


Service Level Agreement Decomposition

The shopping service must

complete in 80 ms

The inventory service must

complete in 20 ms

C* inventory query must

complete in 8 ms

The rates service must complete in

30 ms

C* rates query must complete in

10 ms

A shopping request for a 3-night stay at rack rates for a property that has 15 room types must have a 95 percentile completion time of 80 ms


Roadmap

Cassandra 3.XMaterialized Views & SASI

Row CacheSpark & Search


Final Thoughts

Let Cassandra be CassandraBe Flexible in Consistency

Manage the JoinsGet to Scale

Automate Everything… with care

We’re Hiring!


http://careers.choicehotels.com

• Sr. Cassandra Database Administrator• DevOps Architect• Multiple Java development positions

Now Available!


Cassandra: The Definitive Guide, 2nd EditionCompletely reworked for Cassandra 3.X:

• Data modeling in CQL

• SASI indexes

• materialized views

• lightweight transactions

• DataStax drivers

• New chapters on security, deployment, and integration


Contact us

@JavaBakerag @choicehotels

Choice Hotels International

@jscarp

building a distributed reservation system using cassandra

Software