![Page 1: A Dynamic Caching Mechanism for Hadoop using Memcached Gurmeet Singh Puneet Chandra Rashid Tahir University of Illinois at Urbana Champaign Presenter:](https://reader036.vdocuments.net/reader036/viewer/2022081520/56649d6e5503460f94a50061/html5/thumbnails/1.jpg)
A Dynamic Caching Mechanism for Hadoopusing Memcached
Gurmeet Singh Puneet Chandra Rashid TahirUniversity of Illinois at Urbana Champaign Presenter: Chang Dong
![Page 2: A Dynamic Caching Mechanism for Hadoop using Memcached Gurmeet Singh Puneet Chandra Rashid Tahir University of Illinois at Urbana Champaign Presenter:](https://reader036.vdocuments.net/reader036/viewer/2022081520/56649d6e5503460f94a50061/html5/thumbnails/2.jpg)
Outline
1.Memcached2.Hadoop-Memcached
![Page 3: A Dynamic Caching Mechanism for Hadoop using Memcached Gurmeet Singh Puneet Chandra Rashid Tahir University of Illinois at Urbana Champaign Presenter:](https://reader036.vdocuments.net/reader036/viewer/2022081520/56649d6e5503460f94a50061/html5/thumbnails/3.jpg)
1 Memcached
![Page 4: A Dynamic Caching Mechanism for Hadoop using Memcached Gurmeet Singh Puneet Chandra Rashid Tahir University of Illinois at Urbana Champaign Presenter:](https://reader036.vdocuments.net/reader036/viewer/2022081520/56649d6e5503460f94a50061/html5/thumbnails/4.jpg)
What is memcached briefly?
• memcached is a high-performance, distributed memory object caching system, generic in nature
• It is a key-based cache daemon that stores data and objects wherever dedicated or spare RAM is available for very quick access
• It is a dumb distributed hash table. It does not provide redundancy, failover or authentication. If needed the client has to handle that.
![Page 5: A Dynamic Caching Mechanism for Hadoop using Memcached Gurmeet Singh Puneet Chandra Rashid Tahir University of Illinois at Urbana Champaign Presenter:](https://reader036.vdocuments.net/reader036/viewer/2022081520/56649d6e5503460f94a50061/html5/thumbnails/5.jpg)
Why was memcached made?
• It was originally developed by Danga Interactive to enhance the speed of LiveJournal.com
• It dropped the database load to almost nothing, yielding faster page load times for users, better resource utilization, and faster access to the databases on a memcache miss
• http://www.danga.com/memcached/
![Page 6: A Dynamic Caching Mechanism for Hadoop using Memcached Gurmeet Singh Puneet Chandra Rashid Tahir University of Illinois at Urbana Champaign Presenter:](https://reader036.vdocuments.net/reader036/viewer/2022081520/56649d6e5503460f94a50061/html5/thumbnails/6.jpg)
6
Memcached
![Page 7: A Dynamic Caching Mechanism for Hadoop using Memcached Gurmeet Singh Puneet Chandra Rashid Tahir University of Illinois at Urbana Champaign Presenter:](https://reader036.vdocuments.net/reader036/viewer/2022081520/56649d6e5503460f94a50061/html5/thumbnails/7.jpg)
Where does memcached reside?
• Memcache is not part of the database but sits outside it on the server(s).
• Over a pool of servers
![Page 8: A Dynamic Caching Mechanism for Hadoop using Memcached Gurmeet Singh Puneet Chandra Rashid Tahir University of Illinois at Urbana Champaign Presenter:](https://reader036.vdocuments.net/reader036/viewer/2022081520/56649d6e5503460f94a50061/html5/thumbnails/8.jpg)
Architecture
![Page 9: A Dynamic Caching Mechanism for Hadoop using Memcached Gurmeet Singh Puneet Chandra Rashid Tahir University of Illinois at Urbana Champaign Presenter:](https://reader036.vdocuments.net/reader036/viewer/2022081520/56649d6e5503460f94a50061/html5/thumbnails/9.jpg)
Why use memcached?
• To reduce the load on the database by caching data BEFORE it hits the database
• Can be used for more then just holding database results (objects) and improve the entire application response time
• Feel the need for speed– Memcache is in RAM - much faster then hitting
the disk or the database
![Page 10: A Dynamic Caching Mechanism for Hadoop using Memcached Gurmeet Singh Puneet Chandra Rashid Tahir University of Illinois at Urbana Champaign Presenter:](https://reader036.vdocuments.net/reader036/viewer/2022081520/56649d6e5503460f94a50061/html5/thumbnails/10.jpg)
Why not use memcached?
• Memcache is held in RAM. This is a finite resource.
• Adding complexity to a system just for complexities sake is a waste. If the system can respond within the requirements without it - leave it alone
![Page 11: A Dynamic Caching Mechanism for Hadoop using Memcached Gurmeet Singh Puneet Chandra Rashid Tahir University of Illinois at Urbana Champaign Presenter:](https://reader036.vdocuments.net/reader036/viewer/2022081520/56649d6e5503460f94a50061/html5/thumbnails/11.jpg)
What are the limits of memcached?
• Keys can be no more then 250 characters• Stored data can not exceed 1M (largest typical
slab size)• There are generally no limits to the number of
nodes running memcache• There are generally no limits the the amount of
RAM used by memcache over all nodes– 32 bit machines do have a limit of 4GB though
![Page 12: A Dynamic Caching Mechanism for Hadoop using Memcached Gurmeet Singh Puneet Chandra Rashid Tahir University of Illinois at Urbana Champaign Presenter:](https://reader036.vdocuments.net/reader036/viewer/2022081520/56649d6e5503460f94a50061/html5/thumbnails/12.jpg)
Memcached
![Page 13: A Dynamic Caching Mechanism for Hadoop using Memcached Gurmeet Singh Puneet Chandra Rashid Tahir University of Illinois at Urbana Champaign Presenter:](https://reader036.vdocuments.net/reader036/viewer/2022081520/56649d6e5503460f94a50061/html5/thumbnails/13.jpg)
Memcached Distributed Architecture
![Page 14: A Dynamic Caching Mechanism for Hadoop using Memcached Gurmeet Singh Puneet Chandra Rashid Tahir University of Illinois at Urbana Champaign Presenter:](https://reader036.vdocuments.net/reader036/viewer/2022081520/56649d6e5503460f94a50061/html5/thumbnails/14.jpg)
2 Hadoop-Memcached
![Page 15: A Dynamic Caching Mechanism for Hadoop using Memcached Gurmeet Singh Puneet Chandra Rashid Tahir University of Illinois at Urbana Champaign Presenter:](https://reader036.vdocuments.net/reader036/viewer/2022081520/56649d6e5503460f94a50061/html5/thumbnails/15.jpg)
MapReduce
• Disk access latency 1. Jobs are scheduled on the same node that houses
the associated data 2. data is replicated and placed in numerous ways to
improve throughput and job completion times
![Page 16: A Dynamic Caching Mechanism for Hadoop using Memcached Gurmeet Singh Puneet Chandra Rashid Tahir University of Illinois at Urbana Champaign Presenter:](https://reader036.vdocuments.net/reader036/viewer/2022081520/56649d6e5503460f94a50061/html5/thumbnails/16.jpg)
RAMClouds
• RAMClouds based solely on main memory
![Page 17: A Dynamic Caching Mechanism for Hadoop using Memcached Gurmeet Singh Puneet Chandra Rashid Tahir University of Illinois at Urbana Champaign Presenter:](https://reader036.vdocuments.net/reader036/viewer/2022081520/56649d6e5503460f94a50061/html5/thumbnails/17.jpg)
Contribution
• Propose caching mechanism that finds a balance between the aforementioned approaches.
• Combine data replication and placement algorithms with a proactive fetching and caching mechanism based on Memcached.
![Page 18: A Dynamic Caching Mechanism for Hadoop using Memcached Gurmeet Singh Puneet Chandra Rashid Tahir University of Illinois at Urbana Champaign Presenter:](https://reader036.vdocuments.net/reader036/viewer/2022081520/56649d6e5503460f94a50061/html5/thumbnails/18.jpg)
Design
![Page 19: A Dynamic Caching Mechanism for Hadoop using Memcached Gurmeet Singh Puneet Chandra Rashid Tahir University of Illinois at Urbana Champaign Presenter:](https://reader036.vdocuments.net/reader036/viewer/2022081520/56649d6e5503460f94a50061/html5/thumbnails/19.jpg)
Design
![Page 20: A Dynamic Caching Mechanism for Hadoop using Memcached Gurmeet Singh Puneet Chandra Rashid Tahir University of Illinois at Urbana Champaign Presenter:](https://reader036.vdocuments.net/reader036/viewer/2022081520/56649d6e5503460f94a50061/html5/thumbnails/20.jpg)
Design
• A. Two-Level Greedy Caching Receiver-Only greedy caching policy cache an object locally whenever a node needs it and it
is unavailable in its local cache
Sender-Only greedy caching policy cache an object whenever some other node requests for it
and theobject is in the filesystem but not in the cache
![Page 21: A Dynamic Caching Mechanism for Hadoop using Memcached Gurmeet Singh Puneet Chandra Rashid Tahir University of Illinois at Urbana Champaign Presenter:](https://reader036.vdocuments.net/reader036/viewer/2022081520/56649d6e5503460f94a50061/html5/thumbnails/21.jpg)
Design
• B. Fetching a Cached Block Simultaneous-Requesting Memcached-First
![Page 22: A Dynamic Caching Mechanism for Hadoop using Memcached Gurmeet Singh Puneet Chandra Rashid Tahir University of Illinois at Urbana Champaign Presenter:](https://reader036.vdocuments.net/reader036/viewer/2022081520/56649d6e5503460f94a50061/html5/thumbnails/22.jpg)
Design
• C. Replacement at the Memcached Servers replaces the LRU entry from the hash table and informs the
node that has cached the block.
![Page 23: A Dynamic Caching Mechanism for Hadoop using Memcached Gurmeet Singh Puneet Chandra Rashid Tahir University of Illinois at Urbana Champaign Presenter:](https://reader036.vdocuments.net/reader036/viewer/2022081520/56649d6e5503460f94a50061/html5/thumbnails/23.jpg)
Design
• D. Global Cache Replacement Policy N1 110 115 120 125 N2 50 55 60 65
![Page 24: A Dynamic Caching Mechanism for Hadoop using Memcached Gurmeet Singh Puneet Chandra Rashid Tahir University of Illinois at Urbana Champaign Presenter:](https://reader036.vdocuments.net/reader036/viewer/2022081520/56649d6e5503460f94a50061/html5/thumbnails/24.jpg)
Design
• Prefetching
![Page 25: A Dynamic Caching Mechanism for Hadoop using Memcached Gurmeet Singh Puneet Chandra Rashid Tahir University of Illinois at Urbana Champaign Presenter:](https://reader036.vdocuments.net/reader036/viewer/2022081520/56649d6e5503460f94a50061/html5/thumbnails/25.jpg)
Experiments and Results
![Page 26: A Dynamic Caching Mechanism for Hadoop using Memcached Gurmeet Singh Puneet Chandra Rashid Tahir University of Illinois at Urbana Champaign Presenter:](https://reader036.vdocuments.net/reader036/viewer/2022081520/56649d6e5503460f94a50061/html5/thumbnails/26.jpg)
Experiments and Results
![Page 27: A Dynamic Caching Mechanism for Hadoop using Memcached Gurmeet Singh Puneet Chandra Rashid Tahir University of Illinois at Urbana Champaign Presenter:](https://reader036.vdocuments.net/reader036/viewer/2022081520/56649d6e5503460f94a50061/html5/thumbnails/27.jpg)
Experiments and Results
![Page 28: A Dynamic Caching Mechanism for Hadoop using Memcached Gurmeet Singh Puneet Chandra Rashid Tahir University of Illinois at Urbana Champaign Presenter:](https://reader036.vdocuments.net/reader036/viewer/2022081520/56649d6e5503460f94a50061/html5/thumbnails/28.jpg)
Experiments and Results
![Page 29: A Dynamic Caching Mechanism for Hadoop using Memcached Gurmeet Singh Puneet Chandra Rashid Tahir University of Illinois at Urbana Champaign Presenter:](https://reader036.vdocuments.net/reader036/viewer/2022081520/56649d6e5503460f94a50061/html5/thumbnails/29.jpg)