migrating structured data between hadoop and rdbms
TRANSCRIPT
Who am I?
• Full Stack engineer at Squid Solutions. • Specialised in Big data. • Fun fact: sleeping by myself in my tent on the top of the highest mountains of the world
What I do ?
• Develop of an analytics toolbox. • No setup. No SQL. No compromise. • Generate SQL with a REST API.
It is open source! https://github.com/openbouquet
Topic of today
• You need Scalability? • You need a machine learning toolbox?
Hadoop is the solution.
•But you still need structured data? Our tool provide a solution.
=> We need both!
What does that mean?
• Creation of dataset in Bouquet • Send the dataset to Spark • Enrich inside Spark • Re-injection in original database
How we do it?
User input
Relational DB
SparkBouquet
Create and Send
How does it work?
BouquetRelational DB
Spark
HDFS/ Tachyon
Hive Metastore
User select the data. Bouquet generate the corresponding SQL Code
Kafka
How does it work?
BouquetRelational DB
Spark
HDFS/ Tachyon
Hive Metastore
Data is read from the SQL database
Kafka
How does it work?
BouquetRelational DB
Spark
HDFS/ Tachyon
Hive Metastore
Bouquet creates an avro schema and send the data to Kafka
Kafka
How does it work?
BouquetRelational DB
SparkKafka
HDFS/ Tachyon
Hive Metastore
Kafka Broker(s) receive the data
How does it work?
BouquetRelational DB
Spark
HDFS/ Tachyon
Hive Metastore
Kafka
The hive metastore is updated and the hdfs connectors writes into hdfs
Tachyon?
• Use it as in memory filesystem to replace HDFS. • Interact with Spark using the hdfs plugin. • Transparent from user point of view
How to keep the data structured?Use a schema registry (Avro in Kafka). each schema has a corresponding kafka topic and a distinct hive table.
{ "type": "record", "name": "ArtistGender", "fields" : [ {"name": "count", "type": "long"}, {"name": "gender", "type": "String"]} ] }
Challenges
- Auto creation of topics/table in Hive for each datasets from Bouquet.
- JDBC reads are too slow for something like Kafka. - Issue with types conversion: null is not supported for all cases for example (issue 272 on schema-registry).
- Versions: Kafka 0.9.0, Tachyon 0.7.1, Spark 1.5.2 with HortonWorks 2.3.4 (Dec 2015)
- Hive: Setting the warehouse directory. - In tachyon: Setting up hostname.
Technology choice
• KISS: Kafka + Spark + Tachyon. • Flexible (Hive, In-memory storage) • Easily scalable
• GemFire, SnappyData, Apache Ignite for In-memory storage. • Storm for streaming
Status
Injection DB -> Spark: OK Spark usage: OK Re-injection: In alpha stage.
Re-injection
Two solutions: • Spark user notifies Bouquet that data has changed (using a custom function) • Bouquet pulls the data from spark
We use it for real!
Collaborating with La Poste to be able to use Spark and the re-injection mechanism to use Bouquet and a geographical visualisation.
In the future
• Notebook integration • We got a DSL for bouquet API, we may want to have built-in support spark. • Improve scalability (Bulk Unload and Kafka fine tuning)
QUESTIONS OPENBOUQUET.IO