![Page 1: Duy Hai DOAN - GOTO Conference · @doanduyhai Datastax! • Founded in April 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 400+ employees](https://reader034.vdocuments.net/reader034/viewer/2022042220/5ec6b2e6438b495ec4625cbe/html5/thumbnails/1.jpg)
@doanduyhai
Apache Zeppelin, the missing GUI for your BigData eco-system
Duy Hai DOAN
![Page 2: Duy Hai DOAN - GOTO Conference · @doanduyhai Datastax! • Founded in April 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 400+ employees](https://reader034.vdocuments.net/reader034/viewer/2022042220/5ec6b2e6438b495ec4625cbe/html5/thumbnails/2.jpg)
@doanduyhai
Who Am I ?!Duy Hai DOAN Cassandra technical advocate • talks, meetups, confs • open-source devs (Achilles, …) • OSS Cassandra point of contact
☞ [email protected] ☞ @doanduyhai
2
![Page 3: Duy Hai DOAN - GOTO Conference · @doanduyhai Datastax! • Founded in April 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 400+ employees](https://reader034.vdocuments.net/reader034/viewer/2022042220/5ec6b2e6438b495ec4625cbe/html5/thumbnails/3.jpg)
@doanduyhai
Datastax!• Founded in April 2010
• We contribute a lot to Apache Cassandra™
• 400+ customers (25 of the Fortune 100), 400+ employees
• Headquarter in San Francisco Bay area
• EU headquarter in London, offices in France and Germany
• Datastax Enterprise = OSS Cassandra + extra features
3
![Page 4: Duy Hai DOAN - GOTO Conference · @doanduyhai Datastax! • Founded in April 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 400+ employees](https://reader034.vdocuments.net/reader034/viewer/2022042220/5ec6b2e6438b495ec4625cbe/html5/thumbnails/4.jpg)
What is Apache Zeppelin ?!
Presentation!Architecture!
!
![Page 5: Duy Hai DOAN - GOTO Conference · @doanduyhai Datastax! • Founded in April 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 400+ employees](https://reader034.vdocuments.net/reader034/viewer/2022042220/5ec6b2e6438b495ec4625cbe/html5/thumbnails/5.jpg)
@doanduyhai
Zeppelin Presentation!
5
![Page 6: Duy Hai DOAN - GOTO Conference · @doanduyhai Datastax! • Founded in April 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 400+ employees](https://reader034.vdocuments.net/reader034/viewer/2022042220/5ec6b2e6438b495ec4625cbe/html5/thumbnails/6.jpg)
@doanduyhai
Demo https://github.com/doanduyhai/incubator-zeppelin/tree/ZeppelinPresentation
![Page 7: Duy Hai DOAN - GOTO Conference · @doanduyhai Datastax! • Founded in April 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 400+ employees](https://reader034.vdocuments.net/reader034/viewer/2022042220/5ec6b2e6438b495ec4625cbe/html5/thumbnails/7.jpg)
@doanduyhai
Zeppelin Architecture!
Zeppelin Server
Zeppelin Engine
7
REST
Web
Sock
et
Spark Interpreter Group
Spark SparkSQL
Zeppelin Interpreter
Factory
Tajo Interpreter
Flink Interpreter
Cassandra Interpreter
JVM
JVM
JVM
JVM
JVM
![Page 8: Duy Hai DOAN - GOTO Conference · @doanduyhai Datastax! • Founded in April 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 400+ employees](https://reader034.vdocuments.net/reader034/viewer/2022042220/5ec6b2e6438b495ec4625cbe/html5/thumbnails/8.jpg)
@doanduyhai
What does Zeppelin provide ?!Front-end & display system for free Generic back-end with REST APIs & WebSocket Pluggable interpreters system Task scheduler (à la CRON)
8
![Page 9: Duy Hai DOAN - GOTO Conference · @doanduyhai Datastax! • Founded in April 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 400+ employees](https://reader034.vdocuments.net/reader034/viewer/2022042220/5ec6b2e6438b495ec4625cbe/html5/thumbnails/9.jpg)
Zeppelin UI Layout!
Notebook!Paragraph!
UI elements!
![Page 10: Duy Hai DOAN - GOTO Conference · @doanduyhai Datastax! • Founded in April 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 400+ employees](https://reader034.vdocuments.net/reader034/viewer/2022042220/5ec6b2e6438b495ec4625cbe/html5/thumbnails/10.jpg)
@doanduyhai
Demo https://github.com/doanduyhai/incubator-zeppelin/tree/ZeppelinPresentation
![Page 11: Duy Hai DOAN - GOTO Conference · @doanduyhai Datastax! • Founded in April 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 400+ employees](https://reader034.vdocuments.net/reader034/viewer/2022042220/5ec6b2e6438b495ec4625cbe/html5/thumbnails/11.jpg)
Zeppelin Display System!!
Raw, Table, HTML!Available graphs!
View modes!Dynamic form!Iframe export!
![Page 12: Duy Hai DOAN - GOTO Conference · @doanduyhai Datastax! • Founded in April 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 400+ employees](https://reader034.vdocuments.net/reader034/viewer/2022042220/5ec6b2e6438b495ec4625cbe/html5/thumbnails/12.jpg)
@doanduyhai
Demo https://github.com/doanduyhai/incubator-zeppelin/tree/ZeppelinPresentation
![Page 13: Duy Hai DOAN - GOTO Conference · @doanduyhai Datastax! • Founded in April 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 400+ employees](https://reader034.vdocuments.net/reader034/viewer/2022042220/5ec6b2e6438b495ec4625cbe/html5/thumbnails/13.jpg)
Interpreter system !!
Core interpreters!Third-parties interpreters!
Interpreters conf & usage!
![Page 14: Duy Hai DOAN - GOTO Conference · @doanduyhai Datastax! • Founded in April 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 400+ employees](https://reader034.vdocuments.net/reader034/viewer/2022042220/5ec6b2e6438b495ec4625cbe/html5/thumbnails/14.jpg)
@doanduyhai
Interpreter processing lifecycle!① Receive input commands/data • as raw text
• from form data
② Process the input commands/data by the external back-end ③ Format the response using Zeppelin display system ④ Send response back to the Zeppelin engine
14
![Page 15: Duy Hai DOAN - GOTO Conference · @doanduyhai Datastax! • Founded in April 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 400+ employees](https://reader034.vdocuments.net/reader034/viewer/2022042220/5ec6b2e6438b495ec4625cbe/html5/thumbnails/15.jpg)
@doanduyhai
Core interpreters !!• Spark (Spark core, SparkSQL/DataFrame, PySpark) • Spark core = default (or %spark)
• SparkSQL = %sql
• Shell (%sh)
• Markdown (%md) !
• AngularJS (%angular)
15
![Page 16: Duy Hai DOAN - GOTO Conference · @doanduyhai Datastax! • Founded in April 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 400+ employees](https://reader034.vdocuments.net/reader034/viewer/2022042220/5ec6b2e6438b495ec4625cbe/html5/thumbnails/16.jpg)
@doanduyhai
Third-parties interpreters!• Hive • Phoenix • Tajo • Flink • Ignite • Lens • Cassandra • Geode • PostgreSQL • Kylin
16
![Page 17: Duy Hai DOAN - GOTO Conference · @doanduyhai Datastax! • Founded in April 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 400+ employees](https://reader034.vdocuments.net/reader034/viewer/2022042220/5ec6b2e6438b495ec4625cbe/html5/thumbnails/17.jpg)
@doanduyhai
Interpreter conf & usage https://github.com/doanduyhai/incubator-zeppelin/tree/ZeppelinPresentation
![Page 18: Duy Hai DOAN - GOTO Conference · @doanduyhai Datastax! • Founded in April 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 400+ employees](https://reader034.vdocuments.net/reader034/viewer/2022042220/5ec6b2e6438b495ec4625cbe/html5/thumbnails/18.jpg)
Writing An Interpreter !!
How To!Simple interpreter example (AsciiDoc)!
Complex interpreter example (Cassandra)!
![Page 19: Duy Hai DOAN - GOTO Conference · @doanduyhai Datastax! • Founded in April 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 400+ employees](https://reader034.vdocuments.net/reader034/viewer/2022042220/5ec6b2e6438b495ec4625cbe/html5/thumbnails/19.jpg)
@doanduyhai
Steps to write your own interpreter!
• Create a class that extends Interpreter base class
• Register it in a static block
• Optionnally define default config params
19
static { Interpreter.register("MyInterpreterName", MyClassName.class.getName());
}
static { Interpreter.register("MyInterpreterName", MyClassName.class.getName(), new InterpreterPropertyBuilder() .add("property1", "default value", "Description of property1").build());
}
![Page 20: Duy Hai DOAN - GOTO Conference · @doanduyhai Datastax! • Founded in April 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 400+ employees](https://reader034.vdocuments.net/reader034/viewer/2022042220/5ec6b2e6438b495ec4625cbe/html5/thumbnails/20.jpg)
@doanduyhai
To register your interpreter as default !
• Edit the enum ZeppelinConfiguration.ConfVars
• Add your interpreter FQCN in the property ZEPPELIN_INTERPRETERS
20
![Page 21: Duy Hai DOAN - GOTO Conference · @doanduyhai Datastax! • Founded in April 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 400+ employees](https://reader034.vdocuments.net/reader034/viewer/2022042220/5ec6b2e6438b495ec4625cbe/html5/thumbnails/21.jpg)
@doanduyhai
To register your interpreter in config files!
• Create conf/zeppelin-site.xml from conf/zeppelin-site.xml.template
• Add your interpreter FQCN in the property zeppelin.interpreters
21
<property> <name>zeppelin.interpreters</name> <value>org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter, org.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter, org.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.shell.ShellInterpreter, org.apache.zeppelin.hive.HiveInterpreter,com.me.MyNewInterpreter </value>
</property>
![Page 22: Duy Hai DOAN - GOTO Conference · @doanduyhai Datastax! • Founded in April 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 400+ employees](https://reader034.vdocuments.net/reader034/viewer/2022042220/5ec6b2e6438b495ec4625cbe/html5/thumbnails/22.jpg)
@doanduyhai
Simple AsciiDoc Interpreter!
22
Zeppelin Server
AsciiDoc Interpreter
JVM Zeppelin Engine
Raw Text Block
Raw Text Block
Converted To
HTML
HTML Output
① ②
③ ④
JVM
![Page 23: Duy Hai DOAN - GOTO Conference · @doanduyhai Datastax! • Founded in April 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 400+ employees](https://reader034.vdocuments.net/reader034/viewer/2022042220/5ec6b2e6438b495ec4625cbe/html5/thumbnails/23.jpg)
@doanduyhai
Simple interpreter (AsciiDoc) https://github.com/doanduyhai/incubator-zeppelin/tree/ZeppelinPresentation
![Page 24: Duy Hai DOAN - GOTO Conference · @doanduyhai Datastax! • Founded in April 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 400+ employees](https://reader034.vdocuments.net/reader034/viewer/2022042220/5ec6b2e6438b495ec4625cbe/html5/thumbnails/24.jpg)
@doanduyhai
Cassandra Interpreter Architecture!
24
Cassandra Interpreter
JVM
Display Results as
HTML
① ②
⑤
Zeppelin Server
JVM
Raw Text Block
Raw Text Block
Cassandra Cassandra
Java Driver
③
Async CQL statements
④ Render HTML
⑥
![Page 25: Duy Hai DOAN - GOTO Conference · @doanduyhai Datastax! • Founded in April 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 400+ employees](https://reader034.vdocuments.net/reader034/viewer/2022042220/5ec6b2e6438b495ec4625cbe/html5/thumbnails/25.jpg)
@doanduyhai
Cassandra Interpreter Commands!
25
Native CQL statements SELECT * FROM …; INSERT INTO …; …
Schema commands DESCRIBE TABLE …; DESCRIBE KEYSPACE …; …
Prepared statements Commands
@prepare …; @bind …; @remove_prepared …;
Help command HELP;
Options Commands @consistency …; @retryPolicy …; @fetchSize …;
![Page 26: Duy Hai DOAN - GOTO Conference · @doanduyhai Datastax! • Founded in April 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 400+ employees](https://reader034.vdocuments.net/reader034/viewer/2022042220/5ec6b2e6438b495ec4625cbe/html5/thumbnails/26.jpg)
@doanduyhai
Complex interpreter (Cassandra) https://github.com/doanduyhai/incubator-zeppelin/tree/ZeppelinPresentation
![Page 27: Duy Hai DOAN - GOTO Conference · @doanduyhai Datastax! • Founded in April 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 400+ employees](https://reader034.vdocuments.net/reader034/viewer/2022042220/5ec6b2e6438b495ec4625cbe/html5/thumbnails/27.jpg)
Zeppelin future!!
Roadmap!
![Page 28: Duy Hai DOAN - GOTO Conference · @doanduyhai Datastax! • Founded in April 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 400+ employees](https://reader034.vdocuments.net/reader034/viewer/2022042220/5ec6b2e6438b495ec4625cbe/html5/thumbnails/28.jpg)
@doanduyhai
Roadmap & future!• More graph options (Map viz ZEPPELIN-157)
• Helium project, packaging Zeppelin view, logic (code) & resource into Applications
• Interpreters packaging re-design • ship & compile core interpreters only
• third-parties interpreters can be pulled from repository
• which interpreter is core ? Who will maintain ? Community….
• Integrate security (Apache Shiro, ZEPPELIN-53 )
28
![Page 29: Duy Hai DOAN - GOTO Conference · @doanduyhai Datastax! • Founded in April 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 400+ employees](https://reader034.vdocuments.net/reader034/viewer/2022042220/5ec6b2e6438b495ec4625cbe/html5/thumbnails/29.jpg)
@doanduyhai
Roadmap & future!• Out of incubation state to become 1st class Apache project
29
![Page 30: Duy Hai DOAN - GOTO Conference · @doanduyhai Datastax! • Founded in April 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 400+ employees](https://reader034.vdocuments.net/reader034/viewer/2022042220/5ec6b2e6438b495ec4625cbe/html5/thumbnails/30.jpg)
@doanduyhai
Q & R
! " !
![Page 32: Duy Hai DOAN - GOTO Conference · @doanduyhai Datastax! • Founded in April 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 400+ employees](https://reader034.vdocuments.net/reader034/viewer/2022042220/5ec6b2e6438b495ec4625cbe/html5/thumbnails/32.jpg)
@doanduyhai