agildata - how i learned to stop worrying and evolve with on-demand schemas
TRANSCRIPT
How I Learned to Stop Worrying and Evolve with On-‐Demand Schemas
Dan Lynn, CEO www.agildata.com
[email protected] @danklynn
...I was struck by lightning...and bit by a cobra.
Water is quite Agile
2001: The Agile Manifesto is published Today
Massive Progress
1. Customer satisfaction by rapid delivery of useful software
2. Welcome changing requirements, even late in development
3. Working software is delivered frequently (weeks rather than months)
4. Close, daily cooperation between business people and developers
5. Projects are built around motivated individuals, who should be trusted
6. Face-‐to-‐face conversation is the best form of communication (co-‐location)
7. Working software is the principal measure of progress
8. Sustainable development, able to maintain a constant pace
9. Continuous attention to technical excellence and good design
10. Simplicity—the art of maximizing the amount of work not done—is essential
11. Self-‐organizing teams
12. Regular adaptation to changing circumstance
12 Principles of
Agile Development
http://www.gartner.com/newsroom/id/2823619
Mobile market is putting pressure on development to adopt rapid development
and deployment practices that constantly iterate as expectations change.
Traditional Development Practices Will Fail…. Agile Development is Essential.
August, 2014
But…
Questions?
Changing the database schema late in the development tends to cause wide-‐spread breakages in application software.
-‐-‐ Martin Fowler
http://martinfowler.com/articles/evodb.html
...changing a schema after deployment results in painful data migration
problems
-‐-‐ also Martin Fowler
http://martinfowler.com/articles/evodb.html
…which can be frustrating
The Database Must be Agile
What makes a database Agile?
It’s More Than Schema Migration
Ok, so what makes a database Agile?
Simple
Scalable
Streaming
CREATE STREAM s_customer_orders AS SELECT o.order_id, o.order_amount, c.first_name, c.last_name FROM orders_stream o JOIN customers c ON o.customer_id = c.id;
SQL
How?
Processing must happen next to the data
AgilData Broker
v_people partition 0
SELECT a, b
AgilData Broker
v_people partition 1
SELECT a, b
USE SCHEMA example; CREATE STREAM s_one AS SELECT a, b FROM v_people;
Local execution
Local execution
Processing Must Happen Next To The Data
CREATE SCHEMA example WITH 2 PARTITIONS; CREATE VIEW v_people ( a Person, b INTEGER ) PARTITION BY Modulus(a.id);
AgilData Broker
v_people partition 0
AgilData Broker
v_people partition 1
User-Defined Partitioner
Your data views are Just Snapshots of Streams
http://wearcam.org/dundas-square/2004sep23/cement.jpg
Master SlaveReplicationBinary log
files
Be log-‐oriented
Master Slave
Binary log files
Binary log files
Binary logfiles
Be log-‐oriented
Master Slave
Binary log files
Binary log files
Binary logfiles
Be log-‐oriented
Binary log files
Binary log files
Binary logfiles
0 1 2 3 4 5 6 7 8 9 . .
INSERT INTO employees...
UPDATE employees SET salary = 1 MILLION DOLLARS
Be log-‐oriented
AgilData Broker
Stream Input (local WAL or network)
Stream Output (Partition: 0..n)Stream Output (Partition: 0..n)Stream Output (WAL or in-memory)
Stream Runtime User-Defined Functions, Stream Functions, SELECTs
(Optimized by Execution Planner)
Network Out
Logs Allow Arbitrary Stream Computation
ZookeeperZookeeperZookeeper
AgilData Broker
LEADER - Topic: example Partition: 0
FOLLOWER - Topic: example Partition: 1
AgilData Broker
FOLLOWER - Topic: example Partition: 0
LEADER - Topic: example Partition: 1
Replication
Replication
Logs get you reliability
AgilData Cluster
Distributed Stream Processing
High-PerformanceEmbedded Storage Database Versioning
User-Defined Functions& Extensible Data Types Intelligent Query Planning Job Scheduling
3rd-partydata storesMessage QueuesEvent Streams
Clients
Applications(web apps, APIs, mobile)
Analytics(e.g. Tableau, Spark) Monitoring & Management
The simplest way for enterprises to leverage big data without having to become experts
in a complex and changing toolset.
Dan Lynnwww.agildata.com
[email protected]@danklynn
Come see us at Booth #340!