![Page 1: Accumulo Summit 2014: Open Source Graph Analysis and Visualization powered by Accumulo](https://reader038.vdocuments.net/reader038/viewer/2022100516/547db39bb4795984508b4982/html5/thumbnails/1.jpg)
Open Source Graph Analysis
and Visualization
Powered by Accumulo
+Jeff Kunkle June 12, 2014
![Page 2: Accumulo Summit 2014: Open Source Graph Analysis and Visualization powered by Accumulo](https://reader038.vdocuments.net/reader038/viewer/2022100516/547db39bb4795984508b4982/html5/thumbnails/2.jpg)
is an open source big data analysis and visualization platform powered by Accumulo
![Page 3: Accumulo Summit 2014: Open Source Graph Analysis and Visualization powered by Accumulo](https://reader038.vdocuments.net/reader038/viewer/2022100516/547db39bb4795984508b4982/html5/thumbnails/3.jpg)
Elas%cSearch
Built on Scalable Open Source Tech
Hadoop CDH 4
Accumulo
tesseract CLAVIN CMU Sphinx OpenNLP OpenCV ffmpeg
Storm
Secure Graph
custom code
![Page 4: Accumulo Summit 2014: Open Source Graph Analysis and Visualization powered by Accumulo](https://reader038.vdocuments.net/reader038/viewer/2022100516/547db39bb4795984508b4982/html5/thumbnails/4.jpg)
secure graph a secure graph abstraction layer atop Accumulo
![Page 5: Accumulo Summit 2014: Open Source Graph Analysis and Visualization powered by Accumulo](https://reader038.vdocuments.net/reader038/viewer/2022100516/547db39bb4795984508b4982/html5/thumbnails/5.jpg)
Row ID Column Family
Column Qualifier Value Descrip8on
V[vertex id] V -‐ -‐ Vertex existence and visibility
V[vertex id] EOUT [edge id] [label] Out edges
V[vertex id] VOUT [vertex id] [edge label] Out vertex
V[vertex id] EIN [edge id] [label] In edges
V[vertex id] VIN [vertex id] [edge label] In vertex
V[vertex id] PROP [prop name + key] [prop value] Property
V[vertex id] PROPMETA [prop name + key] [prop meta] Property Metadata
Vertices Table Format
![Page 6: Accumulo Summit 2014: Open Source Graph Analysis and Visualization powered by Accumulo](https://reader038.vdocuments.net/reader038/viewer/2022100516/547db39bb4795984508b4982/html5/thumbnails/6.jpg)
Row ID Column Family
Column Qualifier Value Descrip8on
E[edge id] E -‐ -‐ Edge existence and visibility
E[edge id] VOUT [vertex id] -‐ Out vertex
E[edge id] VIN [vertex id] -‐ In vertex
E[edge id] PROP [prop name + key] [prop value] Property
E[edge id] PROPMETA [prop name + key] [prop meta] Property Metadata
Edges Table Format
![Page 7: Accumulo Summit 2014: Open Source Graph Analysis and Visualization powered by Accumulo](https://reader038.vdocuments.net/reader038/viewer/2022100516/547db39bb4795984508b4982/html5/thumbnails/7.jpg)
Show key features of Lumify
Show how Accumulo is used to implement the features
![Page 8: Accumulo Summit 2014: Open Source Graph Analysis and Visualization powered by Accumulo](https://reader038.vdocuments.net/reader038/viewer/2022100516/547db39bb4795984508b4982/html5/thumbnails/8.jpg)
graph security
sandboxed workspaces 2
1
index security 3
![Page 9: Accumulo Summit 2014: Open Source Graph Analysis and Visualization powered by Accumulo](https://reader038.vdocuments.net/reader038/viewer/2022100516/547db39bb4795984508b4982/html5/thumbnails/9.jpg)
Key Concepts
structure for organizing information (i.e., your data model) Ontology
any “thing” you want to represent (e.g., person, place, event) Entities
a link between two entities (e.g., leader of, works for, sibling of) Relationships
data about an entity (e.g., first name, last name, date of birth) Properties
collection of entities and the relationships between them Graph
![Page 10: Accumulo Summit 2014: Open Source Graph Analysis and Visualization powered by Accumulo](https://reader038.vdocuments.net/reader038/viewer/2022100516/547db39bb4795984508b4982/html5/thumbnails/10.jpg)
graph security
![Page 11: Accumulo Summit 2014: Open Source Graph Analysis and Visualization powered by Accumulo](https://reader038.vdocuments.net/reader038/viewer/2022100516/547db39bb4795984508b4982/html5/thumbnails/11.jpg)
Wallmart (vertex V3)
Row ID Column Family Column Qualifier Visibility Value
V3 V -‐ U
V3 EIN E1 TS Is leader
V3 VIN V1 TS Is leader
V3 EIN E3 S works for
V3 VIN V4 S works for
V3 EOUT E2 U headquartered in
V3 VOUT V2 U headquartered in
V3 PROP name1 U Wallmart
V3 PROP founded1 S 1962-‐01-‐01
![Page 12: Accumulo Summit 2014: Open Source Graph Analysis and Visualization powered by Accumulo](https://reader038.vdocuments.net/reader038/viewer/2022100516/547db39bb4795984508b4982/html5/thumbnails/12.jpg)
User with U, S, and TS visibility
Row ID Column Family Column Qualifier Visibility Value
V3 V -‐ U
V3 EIN E1 TS Is leader
V3 VIN V1 TS Is leader
V3 EIN E3 S works for
V3 VIN V4 S works for
V3 EOUT E2 U headquartered in
V3 VOUT V2 U headquartered in
V3 PROP name1 U Wallmart
V3 PROP founded1 S 1962-‐01-‐01
![Page 13: Accumulo Summit 2014: Open Source Graph Analysis and Visualization powered by Accumulo](https://reader038.vdocuments.net/reader038/viewer/2022100516/547db39bb4795984508b4982/html5/thumbnails/13.jpg)
User with U and S visibility
Row ID Column Family Column Qualifier Visibility Value
V3 V -‐ U
V3 EIN E1 TS Is leader
V3 VIN V1 TS Is leader
V3 EIN E3 S works for
V3 VIN V4 S works for
V3 EOUT E2 U headquartered in
V3 VOUT V2 U headquartered in
V3 PROP name1 U Wallmart
V3 PROP founded1 S 1962-‐01-‐01
![Page 14: Accumulo Summit 2014: Open Source Graph Analysis and Visualization powered by Accumulo](https://reader038.vdocuments.net/reader038/viewer/2022100516/547db39bb4795984508b4982/html5/thumbnails/14.jpg)
User with U visibility
Row ID Column Family Column Qualifier Visibility Value
V3 V -‐ U
V3 EIN E1 TS Is leader
V3 VIN V1 TS Is leader
V3 EIN E3 S works for
V3 VIN V4 S works for
V3 EOUT E2 U headquartered in
V3 VOUT V2 U headquartered in
V3 PROP name1 U Wallmart
V3 PROP founded1 S 1962-‐01-‐01
![Page 15: Accumulo Summit 2014: Open Source Graph Analysis and Visualization powered by Accumulo](https://reader038.vdocuments.net/reader038/viewer/2022100516/547db39bb4795984508b4982/html5/thumbnails/15.jpg)
sandboxed workspaces
![Page 16: Accumulo Summit 2014: Open Source Graph Analysis and Visualization powered by Accumulo](https://reader038.vdocuments.net/reader038/viewer/2022100516/547db39bb4795984508b4982/html5/thumbnails/16.jpg)
Zarka de Mexico Vertex (V3)
Row ID Column Family Column Qualifier Visibility Value
V3 V -‐ U
V3 EIN E1 TS Is leader
V3 VIN V1 TS Is leader
V3 EIN E3 S works for
V3 VIN V4 S works for
V3 EOUT E2 U headquartered in
V3 VOUT V2 U headquartered in
V3 EIN E8 S&WS1 works for
V3 VIN V8 S&WS1 works for
![Page 17: Accumulo Summit 2014: Open Source Graph Analysis and Visualization powered by Accumulo](https://reader038.vdocuments.net/reader038/viewer/2022100516/547db39bb4795984508b4982/html5/thumbnails/17.jpg)
index security
![Page 18: Accumulo Summit 2014: Open Source Graph Analysis and Visualization powered by Accumulo](https://reader038.vdocuments.net/reader038/viewer/2022100516/547db39bb4795984508b4982/html5/thumbnails/18.jpg)
Implemented in ElasticSearch
• Use parent/child document indexing. One document per property.
• Store visibility with indexed docs. • Custom-developed ES filter uses
Accumulo’s visibility evaluation code to filter out documents prior to query eval.
![Page 19: Accumulo Summit 2014: Open Source Graph Analysis and Visualization powered by Accumulo](https://reader038.vdocuments.net/reader038/viewer/2022100516/547db39bb4795984508b4982/html5/thumbnails/19.jpg)
demo
![Page 20: Accumulo Summit 2014: Open Source Graph Analysis and Visualization powered by Accumulo](https://reader038.vdocuments.net/reader038/viewer/2022100516/547db39bb4795984508b4982/html5/thumbnails/20.jpg)
Questions?
learn more at www.lumify.io
Jeff Kunkle @kunklejr
References
SecureGraph • http://securegraph.org • http://youtu.be/JMde_jFDM2M