apache&ignite:& - gridgain systems hazelcast! open source! point solution - imdg!...
TRANSCRIPT
© 2014 GridGain Systems, Inc.
NIKITA IVANOV Founder & CTO @c64hacker
Apache Ignite: Real-‐Time Big Data with In-‐Memory Data Fabric
www.gridgain.com #gridgain
© 2014 GridGain Systems, Inc.
Agenda
• History of GridGain/Apache Ignite • EvoluSon of In-‐Memory CompuSng • In-‐Memory Data Fabric
– Distributed Cluster & Compute – Distributed Data Grid – Distributed Streaming & CEP – Plug-‐n-‐Play Hadoop Accelerator
© 2014 GridGain Systems, Inc.
What is In-‐Memory Compu>ng
• High Performance & Low Latencies • Faster than Disk and Flash • Cost EffecSve • Distributed or Not • Caching, Streaming, ComputaSons • Data Querying – SQL or Unstructured • VolaSle and Persistent • OLAP and OLTP Use Cases
© 2014 GridGain Systems, Inc.
Evolu>on of In-‐Memory Compu>ng
Caching
Distributed Caching
In-‐Memory Data Grids IMDBs
Database IM opSons Hadoop
accelerators
Streaming
BI accelerators
Clustering & Compute Grid
Data Grid Streaming
Hadoop Acceleration
© 2014 GridGain Systems, Inc.
Exis>ng Market is Fragmented
Company Product Proprietary/ Open Source Characteriza>on
Oracle In-Memory Option for Oracle Database Proprietary Cost Option
Oracle Times Ten Proprietary Point Solution IMDB
Oracle Coherence Proprietary Point Solution IMDG
SAP Hana Proprietary Point Solution - IMDB
Microsoft SQL Server 2014 Proprietary Feature Upgrade
DataBricks Apache Spark Open Source Point Solution - Hadoop
VoltDB VoltDB Open Source Point Solution – IMDB
Aerospike Aerospike Open Source Point Solution – NoSQL DB
IBM DB2 with BLU Acceleration Proprietary Feature Upgrade
Software AG Terracotta Open Source Point Solution - IMDG
Hazelcast Hazelcast Open Source Point Solution - IMDG
In-‐Memory Data Fabric: Strategic Approach to IMC
• Supports all Apps
• Open Source – Apache 2.0 • Simple Java APIs • 1 JAR Dependency • High Performance & Scale • Automatic Fault Tolerance • Management/Monitoring • Runs on Commodity Hardware
• Supports existing & new data sources
• No need to rip & replace
Clustering & Compute Grid
Data Grid Streaming
Hadoop Acceleration
© 2014 GridGain Systems, Inc.
© 2014 GridGain Systems, Inc.
Clustering & Compute • Zero Deployment • Pluggable SPI Design • Full Cluster Management • Direct API for MapReduce • Direct API for Fork/Join • Cron-‐like Task Scheduling • State Checkpoints • Early and Late Load Balancing • AutomaSc Failover
© 2014 GridGain Systems, Inc.
Automa>c Cluster Discovery
© 2014 GridGain Systems, Inc.
Closure Execu>on
© 2014 GridGain Systems, Inc.
Closure Execu>on
© 2014 GridGain Systems, Inc.
In-‐Memory Caching and Data Grid
• Distributed In-‐Memory Key-‐Value Store • Replicated and ParSSoned • TBs of data, of any type • On-‐Heap and Off-‐Heap Storage • Backup Replicas / AutomaSc Failover • Distributed ACID TransacSons • SQL queries and JDBC driver • CollocaSon of Compute and Data
© 2014 GridGain Systems, Inc.
Cache Opera>ons
Find a Bug?
© 2014 GridGain Systems, Inc.
Cache Transac>on
© 2014 GridGain Systems, Inc.
Distributed Java Data Structures
• Distributed Map (cache) • Distributed Set • Distributed Queue • CountDownLatch • AtomicLong • AtomicSequence • AtomicReference • Distributed ExecutorService
© 2014 GridGain Systems, Inc.
Client-‐Server vs. Affinity Coloca>on
Client-‐Server Affinity ColocaSon
© 2014 GridGain Systems, Inc.
In-‐Memory Streaming & CEP
• Streaming Data Never Ends • Branching Pipelines • CEP Sliding Windows • Pluggable RouSng • Real Time Analysis • At Least Once Guarantee
© 2014 GridGain Systems, Inc.
Plug-‐n-‐Play Hadoop Accelerator
• Up to 100x AcceleraSon • In-‐Memory NaSve MapReduce
– In-‐Process Data ColocaSon – Eager Push Scheduling
• GGFS In-‐Memory File System – Pure In-‐Memory – Write-‐Through to HDFS – Read-‐Through from HDFS
• Sync and Async Persistence
© 2014 GridGain Systems, Inc.
In-‐Memory Na>ve MapReduce
• In-‐Memory NaSve MapReduce – Zero Code Change – Use exisSng MR code – Use exisSng Hive queries
• No Name Node • No Network Noise • In-‐Process Data ColocaSon • Eager Push Scheduling
© 2014 GridGain Systems, Inc.
DevOps Management and Monitoring
© 2014 GridGain Systems, Inc.
THANK YOU
www.gridgain.com #gridgain