Local setup for HBase, Phoenix and Squirrel

Setting up HBaseExecuting HBase ShellDifferent modes of HBase

Standalone ModeDistributed Mode 

Execution StepsSample QueriesExecution Steps Sample Queries

Setting up SquirrelInstallation StepsTroubleshooting

References

This page aims to serve as a brief guide for doing a local setup for HBase (in standalone mode), Phoenix - for querying over HBase and Squirrel client as additional GUI over Phoenix for SQL querying over HBase

Setting up HBase

HBase can be quickly downloaded and installed using brew formula - Run the command on MAC terminal (and this should complete the installation in /usr/local directory)

brew install hbase

After brew installation is complete - validate the installation by running command hbase version to get output as below

Executing HBase Shell

Once installed, hbase can be executed in shell mode after some configurations. HBase can be run in Stand alone mode (without hadoop on local file system) or in distributed mode (on HDFS). In stand alone mode, HBase also manages its own zookeeper instance.

For running in standalone mode - navigate/open the hbase-site.xml file and make changes as below (hbase-site xml file is located at directory - )/usr/local/Cellar/hbase/1.2.6.1/libexec/conf/

Remove any existing properties from the section of hbase-site.xml and add the below property file

hbase.rootdir file:///Users//hbase

The above property setting essentially creates a directory on local file system where hbase stores all its tables and metadata information

this setup is primarily targeted for MacOS 

1.

2. 3. 4. 5.

a.

b.

Also ensure - for the standalone mode HBASE_MANAGES_ZK is set  - "export HBASE_MANAGES_ZK=true"  in file /libexec/conf/hbase-env.sh

Different modes of HBase

Standalone Mode

This is for local setup and development, executing queries and getting results without much configuration setup

Distributed Mode 

Tables are created in HDFS - this requires Hadoop Installation, and can be configured over clustered setup with multiple region servers and multiple/single zookeeper instances - For production purposes, this is the route/approach that should be adopted

Execution Steps

Once the hbase-site.xml is updated appropriately, perform below steps to run hbase in shell mode

Navigate to hbase installation and move to bin folder - here all executable shell files are available for starting/stopping hbase etc - Path -   /usr/local/Cellar/hbase/1.2.6.1//binExecute start-hbase.sh script from the above directoryOnce the script completes - run jps to ensure HMaster java process is started and runningExecute command in the terminalhbase shellIf the installation was proper the the hbase-site.xml configuration was properly loaded, then the shell should load without any errors

Validate shell starts up and looks as below:

Execute a test command to ensure shell was properly started - enter the command "list" to list all the create tables / system tables etc as below:

Sample Queries

1. a. b.

2.

3.

4. 5. 6. 7.

8.

To create a new table "testTable" with column family "cf1" - execute command - "create 'testtable','cf1'" on shell and validate the output as below.

The newly created table should configuration can be obtained using - "describe testtable" command

 

Setting up Phoenix

Once HBase standalone mode is up and running, its now time to setup Phoenix

Phoenix is nothing but a SQL wrapper over HBase and allows you to read and write to HBase tables through simple SQL like queries.

The SQL queries are underneath converted into HBase scans by Phoenix which results in retrieval or persistence of data from/to HBase tables.

Download Phoenix from apache download mirror - http://www.apache.org/dyn/closer.lua/phoenix/apache-phoenix-4.14.0-HBase-1.4/bin/apache-phoenix-4.14.0-HBase-1.4-bin.tar.gz

Execution Steps 

Once phoenix is downloaded, unzip the tar/zip file to a directory at some location.

Copy / Update the required JARS from phoenix installation to HBase lib Navigate to Phoenix install folder and copy "phoenix-4.14.0-HBase-1.2-server.jar" Paste this JAR to HBase lib directory - /usr/local/Cellar/hbase/1.2.6.1/libexec/lib

Add an environment variable HBASE_CONF_DIR - this should point to path of conf folder in hbase installation  - like /usr/local/Cellar/hbase/1.2.6.1/libexec/conf/  (this is used by sqlline python file for running phoenix command line)

Also update the hbase-site.xml by adding 2 more property for mapping hbase schema to phoenix

phoenix.schema.isNamespaceMappingEnabled true phoenix.schema.mapSystemTablesToNamespace true

Stop and Start HBase once againNavigate to Phoenix install directory and move to bin folder - here all python scripts to start SQL query terminal are availableExecute - ./sqlline.py localhostOnce the command completes - Phoenix terminal should be up and running (if there are any errors - go through the above steps and check if mentioned steps were followed and changes were made)The phoenix terminal should look like this

It should be kept in mind that while downloading Phoenix, the supported version of HBase should be kept in mind - So first retrieve HBase version and depending on that download Phoenix version - For HBase 1.2 - Phoenix 4.14 is the supported version

http://www.apache.org/dyn/closer.lua/phoenix/apache-phoenix-4.14.0-HBase-1.4/bin/apache-phoenix-4.14.0-HBase-1.4-bin.tar.gzhttp://www.apache.org/dyn/closer.lua/phoenix/apache-phoenix-4.14.0-HBase-1.4/bin/apache-phoenix-4.14.0-HBase-1.4-bin.tar.gz

8.

9. Execute a test query on phoenix to ensure the setup was successful - run command " " to list all the tables available - (this !tableslists down all system tables as well as all tables that were created using phoenix)

Sample Queries

Creating a simple table in Phoenix is through normal SQL Queries - 

ex - to create a student table with fields = id (Integer Primary key), birthDate (Date), name ( varchar), age(Integer) is as below

create table student(id INTEGER NOT NULL PRIMARY KEY, birthdate DATE, name varchar(50), age INTEGER);

Execute above statement and listing down tables should now list the newly created student table as well

Setting up Squirrel

After the installation of HBase and Phoenix - Squirrel which is a GUI client for writing SQL queries can be setup for local execution of queries and development/testing of scripts.

Its a generic SQL client which supports various Databases that have JDBC drivers like - MySQL Postgres etc.

1. 2. 3.

a. b.

c. 4. 5.

6.

Squirrel itself is a Java application.

Installation Steps

Download Squirrel client from - http://squirrel-sql.sourceforge.net/Follow the Squirrel Install guide - http://squirrel-sql.sourceforge.net/paper/GettingStartedusingtheSQuirreLSQLClient.pdfCopy the Phoenix client jar from phoenix install location to Squirrel Resources directory

Copy " " from Phoenix install directoryphoenix-4.14.0-HBase-1.2-client.jarPaste this to Squirrel resources directory - Navigate to squirrel icon in Applications folder - Right click on Squirrel Icon and click Show Package Contents

Navigate to /Contents/Resources/java/lib and paste the copied JAROpen Squirrel SQL Client and setup the Phoenix driver and alias Open Phoenix alias (created in step 4 above) and click "Connect" in order to connect to existing instance of HBase

Once connected - it should display all the tables from HBase 

http://squirrel-sql.sourceforge.net/http://squirrel-sql.sourceforge.net/paper/GettingStartedusingtheSQuirreLSQLClient.pdf

6.

1.

a. b. c.

2.

a.

b. c.

1. 2. 3.

Troubleshooting

On MacOS - a lot of times - Squirrel does not work out of the box - i.e. the launcher does not work or show up the application as its unable to resolve the installation path of Squirrel - To fix the issue 

Navigate to Squirrel icon in Applications folder -  Right click on Squirrel Icon and click Show Package ContentsOpen - /MacOS/squirrel-sql.shSearch for SQUIRREL_SQL_HOME and update the value to "Applications/SQuirreLSQL.app/Contents/Resources/Java"

While configuring driver for Phoenix, a lot of times its unable to find out the apt hbase-site xml configuration due to which it reports errors for schema namespace mapping properties - To fix the issue - update the HBase client jar to include the hbase-site.xml file in the jar and then refer this jar while creating phoenix driver in Squirrel

If while connecting to HBase instance from Squirrel - it complains about namespace mapping schema properties, then chances are there that the desired hbase-site.xml properties are not being loaded / correctly obtained by Squirrel client - hence in order to do that - package the HBase client jar with hbase-site xmlExecute the following command (After copying the required hbase-site xml to the Phoenix HBase client jar directory) - jar uf phoenix-4.7.0.2.5.3.0-37-client.jar hbase-site.xml Copy the new jar and paste it to the Squirrel Java lib directory and restart Squirrel Now reconnect with the already set up alias -It should now successfully connect and show all the HBase tables

References

Phoenix Grammar - https://phoenix.apache.org/language/index.htmlSquirrel Install - http://squirrel-sql.sourceforge.net/Apache HBase documentation - https://hbase.apache.org/book.html

Before executing / connecting to Squirrel SQL ensure that HBase is up and running

https://phoenix.apache.org/language/index.htmlhttp://squirrel-sql.sourceforge.net/https://hbase.apache.org/book.html

Local setup for HBase, Phoenix and SquirrelSetting up HBaseExecuting HBase ShellDifferent modes of HBaseStandalone ModeDistributed Mode 

Execution StepsSample QueriesExecution Steps Sample Queries

Setting up SquirrelInstallation StepsTroubleshooting

References