Hands-on Introduction& Hackathon Kickoff

Ashvin Agrawal William Markito@william_markito@aasoj

Powered by

Pivotal Open Source Hub (POSH)

(incubating)

• Hackathon Details• Apache Geode Introduction

• History• Key features and components• Roadmap

• Hands-on lab• Build & run• Starting a cluster• Using docker for clustering• Your first app

• Q&A

Agenda

Hackathon details

Powered by

Pivotal Open Source Hub (POSH)

http://ambitious-apps.challengepost.com/

Introduction

A distributed, memory-based data management platform for data oriented apps that need:• high performance, scalability, resiliency and continuous

availability• fast access to critical data set• location aware distributed data processing• event driven data architecture

Introduction

One size fits all ?

Cost of sorting is nlog(n)

One size fits all ?

Cost of sorting is nlog(n)

• Data quality and quantity differences• Eventual consistency• Response time expectation • Scalability challenges: disk, memory,

network and external systems

• 1000+ systems in production (real customers)• Cutting edge use cases

Incubating… but rock solid

• 17 billion records in memory • GE Power & Water's Remote Monitoring & Diagnostics Center

• 3 TB operational data in-memory, 400 TB archived • China Railways

• 4.6 Million transactions a day / 40K transactions a second • China Railways

Incubating… but rock solid

• Performance optimized persistence

• Configurable consistency

• Elastic capacity

• Latency minimizing distribution

• Heterogenous deployment

Designed for High Performance

+/-

L2 ~10 ns, memory ~100 ns, network <1ms, disk ~10ms

• Cache• Region• Member• Client Cache• Functions• Listeners

Concepts

• Cache

• In-memory storage and management for your data

• Configurable through XML, Spring, Java API or CLI

• Collection of Region

Concepts

• Region

• Distributed java.util.Map on steroids (Key/Value)

• Consistent API regardless of where or how data is stored

• Observable (reactive)

• Highly available, redundant on cache Member (s).

Concepts

• Region

• Local, Replicated or Partitioned

• In-memory or persistent

• Redundant

• LRU

• Overflow

Concepts

LOCALLOCAL_HEAP_LRULOCAL_OVERFLOWLOCAL_PERSISTENTLOCAL_PERSISTENT_OVERFLOWPARTITIONPARTITION_HEAP_LRUPARTITION_OVERFLOWPARTITION_PERSISTENTPARTITION_PERSISTENT_OVERFLOWPARTITION_PROXYPARTITION_PROXY_REDUNDANTPARTITION_REDUNDANTPARTITION_REDUNDANT_HEAP_LRUPARTITION_REDUNDANT_OVERFLOWPARTITION_REDUNDANT_PERSISTENTPARTITION_REDUNDANT_PERSISTENT_OVERFLOWREPLICATEREPLICATE_HEAP_LRUREPLICATE_OVERFLOWREPLICATE_PERSISTENTREPLICATE_PERSISTENT_OVERFLOWREPLICATE_PROXY

• Persistent Regions

• Durability

• WAL for efficient writing

• Consistent recovery

• Compaction

Concepts

Server 1 Server N

• Member

• A process that has a connection to the system

• A process that has created a cache

• Embeddable within your application

Concepts

Client

Locator

Server

• Client cache

• A process connected to the Geode server(s)

• Can have a local copy of the data

• Can be notified about events on the servers

Concepts

• Functions

• Used for distributed concurrent processing (Map/Reduce, stored procedure)

• Highly available

• Data oriented

• Member oriented

Concepts

Concepts

• Functions

• Listeners

• CacheWriter / CacheListener

• AsyncEventListener (queue / batch)

• Parallel or Serial

• Conflation

Concepts

Hands on

• Clone & Build

Hands-on: Build & run

git clone https://github.com/apache/incubator-geodecd incubator-geode./gradlew build -Dskip.tests=true

• Start a server

cd gemfire-assembly/build/install/apache-geode ./bin/gfsh gfsh> start locator --name=locator gfsh> start server --name=server gfsh> create region --name=myRegion --type=REPLICATE

Hands-on: Docker

&

• Containers• FreeBSD Jails (2000)• Solaris Zones (2004)

• Docker (2013)

• Operating system level virtualization• Isolated user space instances

* https://linuxcontainers.org/

Hands-on: Docker

Container vs VM

“..while the hypervisor abstracts the entire device, containers just abstract the operating system kernel"

Hands-on: Docker & Compose

• Single instance

docker run -it apachegeode/geode:nightly gfsh

• Cluster

docker-compose up

• Scale

docker-compose scale server=3

Hands-on: Application

• Teeny URL• Fast response time• Statistics

• Hits• User agent ?• IPs ?

• URL will last for 5 minutes

• Distribute data & load• Highly scalable

createURL

getURLstats

• HDFS Persistence• Off-heap memory storage• Lucene Search• Spark Integration• Cloud Foundry service

Roadmap

• Code• New features• Bug fixes• Writing tests

• Documentation• Wiki• Web site• User guide

How to Contribute

• Community• Join the mailing list

• Ask or answer • Join our HipChat• Become a speaker• Finding bugs• Testing an RC/Beta

• JIRAhttps://issues.apache.org/jira/browse/GEODE

• Wikicwiki.apache.org/confluence/display/GEODE

• GitHubhttps://github.com/apache/incubator-geode

• Mailing listsmail-archives.apache.org/mod_mbox/incubator-geode-dev/

Links

Thank you

http://geode.incubator.apache.org

https://github.com/Pivotal-Open-Source-Hub