Download - In Memory Data Grids, Demystified!
![Page 2: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/2.jpg)
Agenda
• Why IMDG?• Brief History• How It Works– Data model & placement– HA and fault tolerance – Consistency – Internals
![Page 3: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/3.jpg)
Why IMDG?
Today, more than ever, there are many choices when it comes to storing your data
![Page 4: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/4.jpg)
® Copyright 2011 Gigaspaces Ltd. All Rights Reserved
4
But There Many
Solutions
![Page 5: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/5.jpg)
® Copyright 2011 Gigaspaces Ltd. All Rights Reserved
5
Just A Few Years Back
![Page 6: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/6.jpg)
So Why Indeed??
![Page 7: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/7.jpg)
The Need for Speed, In
Real Time…
![Page 8: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/8.jpg)
Some Facts
![Page 9: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/9.jpg)
Memory will always be faster
than disk (usually by orders of
magnitude)
![Page 10: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/10.jpg)
Recent Survey
![Page 11: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/11.jpg)
67%
The ratio of IT managers that think that real time analysis is the biggest challenge for big data implementations
![Page 12: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/12.jpg)
40%
• Plan to use in memory technologies for big data projects.• Only 32%
mentioned Hadoop
![Page 13: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/13.jpg)
Stream Processing
![Page 14: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/14.jpg)
Hell, Even Gartner Thinks So
“In memory computing (IMC) … provides transformational opportunities. The execution of
certain-types of hours-long batch processes can be squeezed into minutes or even seconds …
Millions of events can be scanned in a matter of a few tens of millisecond to detect correlations and patterns
pointing at emerging opportunities and threats "as things happen.”
![Page 15: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/15.jpg)
And nowadays
HW and SW just makes it a whole lot
cheaper
![Page 16: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/16.jpg)
Some Common Use Cases
![Page 17: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/17.jpg)
Fast, Transactional Data Access
• Inventory management • Financial
reference data• Real time
transactional data
![Page 18: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/18.jpg)
Real Time Stream
Processing
• Fraud Detection• Click Stream
Analysis • Real time
analytics • Continuous
calculation
![Page 19: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/19.jpg)
Heavyweight Offline
Calculations
• Trade Reconciliation • Pattern analysis
and detection• Number crunching
![Page 20: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/20.jpg)
Caching
• Database offloading • Content heavy
websites
![Page 21: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/21.jpg)
The Evolution of Data Grids
![Page 22: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/22.jpg)
First There Were Local Caches
CacheIn process cachingof Key->Value data
structure
Distribute CachePartitioned cache
nodes
IMDGPartitioned system
of record
IMDG.next()
Good for repetitive-data reads
Limited in capacity
Doesn’t handle write-heavy scenarios
Reads are only part latency path
![Page 23: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/23.jpg)
Then Came Distributed Caches
CacheIn process cachingof Key->Value data
structure
Distribute CachePartitioned cache
nodes
IMDGPartitioned system
of record
Increased Capacity
Still no support for write-heavy scenarios
Limited to ID-based reads
Reads are only part latency path
IMDG.next()
![Page 24: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/24.jpg)
In Memory Data Grids
CacheIn process cachingof Key->Value data
structure
Increased capacity
Write scalability
Can serve as system of record with querying & transaction semantics
Still limited in capacity
Latency can come from other parts of your app
Distribute CachePartitioned cache
nodes
IMDGPartitioned system
of record
IMDG.next()
![Page 25: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/25.jpg)
How It Works
![Page 26: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/26.jpg)
Data Models
![Page 27: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/27.jpg)
27
Data Placement – Fixed Hashing
hash(key) % #nodes
![Page 28: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/28.jpg)
28
Fixed Hashing - HA
hash(key) % #nodes
![Page 29: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/29.jpg)
29
Fixed Hashing – Scaling
Source: http://www.griddynamics.com/distributed-algorithms-in-nosql-databases/
![Page 30: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/30.jpg)
30
Data Placement – Consistent Hashing
Source: http://www.griddynamics.com/distributed-algorithms-in-nosql-databases/
![Page 31: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/31.jpg)
31
Data Placement – Consistent Hashing
Source: http://www.griddynamics.com/distributed-algorithms-in-nosql-databases/
![Page 32: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/32.jpg)
32
Data Placement – Consistent Hashing
Source: http://www.griddynamics.com/distributed-algorithms-in-nosql-databases/
![Page 33: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/33.jpg)
33
Data Placement – Consistent Hashing
Source: http://www.griddynamics.com/distributed-algorithms-in-nosql-databases/
![Page 34: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/34.jpg)
34
Data Placement – Consistent Hashing
Source: http://www.griddynamics.com/distributed-algorithms-in-nosql-databases/
![Page 35: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/35.jpg)
Data Consistency
Since we’re dealing with distributed data, consistency cannot be taken for granted• Read after write • Read after read • Write-write consistency
![Page 36: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/36.jpg)
Solution 1: Single
Master
![Page 37: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/37.jpg)
Solution 2: Read/Write Quorums
![Page 38: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/38.jpg)
Some More Concerns
• Transactions• Querying • Failure detection • Leader election • Persistency • Interoperability
![Page 39: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/39.jpg)
IMDG.next()
Using IMDG for messaging, BL
![Page 40: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/40.jpg)
IMDG.next()
SSD FTW!
![Page 41: In Memory Data Grids, Demystified!](https://reader036.vdocuments.net/reader036/viewer/2022062511/54b71ca04a7959af718b472e/html5/thumbnails/41.jpg)
Thank You!
docs.gigaspaces.com