on hbase integration testing
If you can't read please download the document
TRANSCRIPT
Secure HBase
On HBase Integration Testing
Andrew Purtell, Hadoop Group at Trend [email protected]
Introducing Apache Bigtop
(and other customdistros)
Introducing Apache Bigtop
A new project in incubation with committers from Cloudera, Ubuntu, and others
The goal is to do for Hadoop what Debian did for Linux
Build and packaging infrastructureMakefile based (yes!)
RPM and DEB packaging targets
Deployment infrastructurePuppet
Integration test infrastructureiTest
Introducing Apache Bigtop
A new project in incubation with committers from Cloudera, Ubuntu, and others
The goal is to do for Hadoop what Debian did for Linux
Build and packaging infrastructureMakefile based (yes!)
RPM and DEB packaging targets
Deployment infrastructurePuppet
Integration test infrastructureiTest
This seems like it could be useful
iTest
A framework for integration testing
Provides a high-level abstraction for querying and driving OS level packaging systems during full system integration testsInstall package
Run tests
Check results
Remove package
Start/stop SysV services
Transparent shell-outs written in Java and GroovyConvenient to write tests that are scripting (Groovy) layered over command line invocations ("hadoop jar ...")
Helper functions for unpacking data from JAR files, filesystem paths, classpath, etc.
HBASE-6201 (and friends)
From Enis Sztutar
Add ability to run all integration/system tests locally or on a given cluster
With 'mvn verify' though Stack suggests this hasn't been possible to date using Maven machinery alone
Or, by importing and using iTest (and exposing this also to BigTop) I personally think this is promising
'mvn verify'
iTest
MiniCluster
Actual cluster
HBASE-6201 (and friends)
Convert candidate unit tests that run on a full MiniCluster into integration tests that can run on a real cluster
TestAcidGuarantees
TestAtomicOperation
TestRegionBalancing
TestFullLogReconstruction
TestMasterFailover
TestImportExport
TestRollingRestart
TestMultiVersions
TestKeepDeletes
TestFromClientSide
TestSplitTransactionOnCluster
... and more
HBASE-6201 (and friends)
Add more tests, especially, long running ingestion tests and chaos monkey
LoadTestTool
YCSB
GoraCI
Resurrect Gremlins? https://github.com/toddlipcon/gremlins
Going beyond HBASE-6201
Add full system application test cases
I have a Heritrix / hbase-writer based webtable thingy ?
JGit backed by HBase https://github.com/apurtell/jgit-hbase ?
Canned queries for Titan (http://thinkaurelius.github.com/titan/ ) or something that uses Apache Giraph's HBase storage adapter ?
Canned data and scripts for Apache Pig and Hive ?
Add Kerberos setup and usage helper functions to iTestso then we can finesse integration testing with security enabled at the Hadoop, HBase, and/or ZooKeeper layers
End. Discuss.