Securely explore your data

ZOOKEEPER,ACCUMULO AND YOU

Michael Allen

Architect

Sqrrl Data, Inc.

THE CASE OF THE DEAD TSERVER

Why when I close my laptop does

my tablet server die?

ACCUMULO CLUSTER

Worker

Hadoop

DataNode

Accumulo

Tserver

Worker

Hadoop

DataNode

Accumulo

Tserver

Accumulo

Master

Zookeeper

Cluster

Hadoop

NameNod

e

ZOOKEEPER CLUSTER

Zookeeper

Cluster

ZOOKEEPER CLUSTER

Zookeeper

Cluster

ZOOKEEPER CLUSTER

Zookeeper

Node

Zookeeper

Node

Zookeeper

Node

ZOOKEEPER CLUSTER

Zookeeper

Node

Zookeeper

Node

Zookeeper

Node

Zookeeper

Node

Zookeeper

Node

WHAT'S ZOOKEEPER GOOD AT?

Small amounts of data

Consistent across cluster

Fast reads

(in memory)

Total order of operations

Zab

ZOOKEEPER DATA

Zookeeper Node

ZOOKEEPER DATA

Zookeeper Node

/

/accumulo

/accumulo/instances

ZOOKEEPER DATA

Zookeeper Node

/

/accumulo

/accumulo/instances

834c234-cd2731

ZOOKEEPER CLIENTS

Zookeeper Node

Zookeeper

Client

2181/tcp

Random long client ID and random password

Keep alive ping

EPHEMERAL NODES

Zookeeper Node

Ephemeral node

SEQUENTIAL NODES

Zookeeper Node

5

Sequential nodes

6

WATCHES

Zookeeper Node

5 6

ACCUMULO DATA IN ZOOKEEPER

/

accumulo

cf7c5ecd-0c3c-4d5c-aed2-36d9ca076976

user

monitor

root_tablet

gc table_locks hdfs_reservationsnamespaces recoveryfate

tservers tables

next_filetracers

config

dead bulk_failed_copyqmasters

instances

tservers

zlock-0000000000

server1.companyco.com:9997

TSERVERS

zlock-0000000000

server2.compnyco.com:9997

config

gc.cycle.delay

CONFIG

general.rpc.timeout general.server.message.size.max

tables

+r

TABLES

!0 1 2 3 4

state conf flush-id compact-id compact-cancel-id name namespace

root_tablet

last_location

ROOT_TABLET

location dir walogs

users

root

USERS

mallen jvines afuchs

USERS

USERS

users

root

ZOOKEEPER ACLS

mallen jvines afuchs

'digest,'accumulo:SkvnZlrIQ19GNd7eLDXGKg0Esgw=: cdrwa

DIGEST SCHEME REALLY MEANS PASSWORDS

UH OH...

I forgot the password I used!

But...

I do have access to zkServer.sh

ROOTING YOUR ZOOKEEPER

1. Create an identity

ZK-DIGEST.SH#!/bin/bash

if [ -z ${ZOOKEEPER_HOME} ]; then

echo "Set \$ZOOKEEPER_HOME before running this script"

exit 4747

fi

if [ -z ${JAVA_HOME} ]; then

echo "Set \$JAVA_HOME before running this script"

exit 4747

fi

if [ $# -eq 0 ]; then

echo "usage: zk-digest.sh <digest string>"

echo ""

echo " Utility to produce authentication digests, such as you might see in ZooKeeper node ACL entries"

echo ""

echo " Example: zk-digest.sh sqrrl:secret"

exit 4747

fi

ZK_CLASSPATH="\

${ZOOKEEPER_HOME}/build/classes:\

${ZOOKEEPER_HOME}/build/lib/*.jar:\

${ZOOKEEPER_HOME}/lib/slf4j-log4j12-1.6.1.jar:\

${ZOOKEEPER_HOME}/lib/slf4j-api-1.6.1.jar:\

${ZOOKEEPER_HOME}/lib/netty-3.2.2.Final.jar:\

${ZOOKEEPER_HOME}/lib/log4j-1.2.15.jar:\

${ZOOKEEPER_HOME}/lib/jline-0.9.94.jar:\

${ZOOKEEPER_HOME}/zookeeper-3.4.5.jar:\

${ZOOKEEPER_HOME}/src/java/lib/*.jar:\

${ZOOKEEPER_HOME}/conf\

"

${JAVA_HOME}/bin/java -Dzookeeper.log.dir="." \

-Dzookeeper.root.logger="INFO,CONSOLE" \

-cp "${ZK_CLASSPATH}" \

-Dcom.sun.management.jmxremote \

-Dcom.sun.management.jmxremote.local.only=false \

org.apache.zookeeper.server.auth.DigestAuthenticationProvider $*

ROOTING YOUR ZOOKEEPER

1. Create an identityzk-digest.sh super:secretsuper:secret->super:lK75jTNcA+U9vtVEw5vB51mj/w4=

ROOTING YOUR ZOOKEEPER

1. Create an identity

2. Edit zk-server.sh

ROOTING YOUR ZOOKEEPER

1. Create an identity

2. Edit zk-server.shnohup $JAVA "-Dzookeeper.log.dir=${ZOO_LOG_DIR}" "-Dzookeeper.root.logger=${ZOO_LOG4J_PROP}" \-cp "$CLASSPATH" $JVMFLAGS $ZOOMAIN "$ZOOCFG" > "$_ZOO_DAEMON_OUT" 2>&1 < /dev/null &

ROOTING YOUR ZOOKEEPER

1. Create an identity

2. Edit zk-server.shnohup $JAVA "-Dzookeeper.log.dir=${ZOO_LOG_DIR}" "-Dzookeeper.root.logger=${ZOO_LOG4J_PROP}" \

"-Dzookeeper.DigestAuthenticationProvider.superDigest=super:lK75jTNcA+U9vtVEw5vB51mj/w4= \"-cp "$CLASSPATH" $JVMFLAGS $ZOOMAIN "$ZOOCFG" > "$_ZOO_DAEMON_OUT" 2>&1 < /dev/null &

ROOTING YOUR ZOOKEEPER

1. Create an identity

2. Edit zk-server.sh

3. Reboot Zookeeper

ROOTING YOUR ZOOKEEPER

THE CASE OF THE DEAD TSERVER

Why when I close my laptop does

my tablet server die?

THE CASE OF THE DEAD TSERVER

tservers

zlock-0000000000

server1.companyco.com:9997

THE CASE OF THE DEAD TSERVER

tservers

zlock-0000000000

server1.companyco.com:9997

QUESTIONS?