apache accumulo and cloudera

Click here to load reader

Post on 27-Jan-2015

107 views

Category:

Technology

3 download

Embed Size (px)

DESCRIPTION

 

TRANSCRIPT

  • 1. ApacheAccumuloandCloudera Hadoop-DC,July2013 JoeyEcheverria|Director,FederalFTS joey@cloudera.com|@fwio 2013Cloudera,Inc.AllRightsReserved. 1

2. ApacheAccumuloandCloudera HADOOP101 2 3. OperaNngSystems Manageandschedulemachineresources CPU RAM Memory ProvideabstracNonsandAPIs Files=streamofbytes Process=instrucNons+privatememoryspace 3 4. DistributedOperaNngSystem Samething,butoveraclusterofnetworkedservers AddiNonalconcerns: Inter-processandinter-machinecommunicaNon Datalocality Dataavailability Dataprocessingavailability 4 5. Hadoop DefactoDistributedOperaNngSystem ApacheHDFS ApacheMapReduceandApacheYARN 5 6. Ecosystem 6 KeyValueStores HighLevelBatchLanguages LowLatencySQLEngineGraphProcessing 7. Cloudera 7 8. CDHHistory 8 CDH1 *HDFS *MR *Hive *Pig CDH2 *HDFS *MR *Hive *Pig CDH3 *HDFS *MR *Hive *Pig *Flume *HBase Hue *Mahout *Oozie *Sqoop *Whirr *Zookeeper *Avro CDH4 *HDFS *MR *YARN *Hive *Pig *Flume *HBase Hue *Mahout *Oozie *Sqoop *Whirr *Zookeeper *Avro DataFu HCatalog Impala *Solr *BigTop Sentry 9. ApacheAccumuloandCloudera ACCUMULO101AND201 9 10. BigTable 10 11. AccumuloDataModel MulJ-dimensionalsortedmap row id -> [ family -> [ qualifier -> [ visibility -> [ timestamp -> value ] ] ] ] 11 12. AccumuloStorageModel key->value key= column= 12 Key Value RowID Column Timestamp Family Qualier Visibility 13. 13 14. OtherConcerns Write-aheadlog Tabletserverfailurehandling Versioning Iterators Cell-levelsecurity 14 15. ApacheAccumuloandCloudera PROJECTHISTORY 15 16. Pre-Apache 16 17. Apache 17 18. RelaNonshiptoHadoopReleases 1.3.x->Hadoop0.20.2 1.4.x->Hadoop0.20.2,Hadoop0.20.203 1.5.x->Hadoop1.0.4,Hadoop2.0.4-alpha 18 19. AccumuloandClouderaReleases Accumulo1.3.x,1.4.x,and1.5.xallworkwithCDH3 Accumulo1.5.xshouldworkwithCDH4 LimitedtesNng 19 20. ApacheAccumuloandCloudera ANNOUNCEMENT 20 21. ApacheAccumuloandCloudera CLOUDERASUPPORTOFAPACHE ACCUMULOONCDH4 21 22. ApacheAccumuloandCloudera DEMO 22 23. SystemLogs Id UniqueidforanacNon Timestamp TimetheacNonoccured Actor UserorsystemperformingtheacNon AcNon TheacNontaken Object TheobjectoftheacNon Info FreeforminformaNon(e.g.success/failure,alributevalue,etc.) 23 24. AcNons created_user deleted_user set_password logged_in logged_out read modied 24 25. Roles system Anyuseronthesystem admin Administrators audit Auditors 25 26. AccumuloDataModel 26 Key Value RowID Column Timestamp Family Qualier Visibility - : 27. ApacheAccumuloandCloudera DEMO 27 28. LogsDemo 28 Rowkey Column Visibility Value 201307241535-1 root:created_user:sean audit succeeded 201307241535-1 root:set_password:sean admin&audit password 201307241537-2 sean:logged_in:host system succeeded 201307241538-3 sean:read:/tmp/a audit succeeded 201307241539-4 sean:modied:/tmp/a audit failed 201307241540-5 sean:logged_out:host system succeeded 29. ApacheAccumuloandCloudera VERSIONSREDUX 29 30. Recap Accumulo1.3.x,1.4.x,and1.5.xallworkwithCDH3 Accumulo1.5.xshouldworkwithCDH4 30 31. ClouderaSupport Naturally,Clouderahastestedandpackaged Accumulo1.5 But1.5isratherbleedingedge So,weinsteadbackportedHadoop2.0supportfrom 1.5onto1.4.3 31 32. ApacheAccumuloandCloudera ECOSYSTEMINTEGRATION 32 33. ApacheNutch 33 34. ApachePig 34 35. ApacheAccumuloandCloudera DEMO 35 36. ApacheAccumuloandCloudera NEXTSTEPS 36 37. Recap Whatsavailabletoday BetareleaseofAccumulo1.4.3onCDH4.3 BetareleaseofAccumulo1.4.3PigintegraNon Semi-privatebeta Contactme(joey@cloudera.com)ifyoureinterestedin tryingoutthebits 37 38. FutureIdeas(notpromises;) ClouderaManagerintegraNon FlumeintegraNon SqoopintegraNon HiveintegraNon ImpalaintegraNon 38 39. Whatnext? DownloadHadoop! CDHavailableatwww.cloudera.com Clouderaprovidespre-loadedVMs hlps://ccp.cloudera.com/display/SUPPORT/Cloudera +QuickStart+VM Reachouttome(joey@cloudera.com)ifyouwantto tryouttheAccumulobeta InstrucNonstoreplicatethedemospending 40. Mypersonalpreference ClouderaManager hlps://ccp.cloudera.com/display/SUPPORT/Downloads Freeuptounlimitednodes! 41. ShoutOut JasonTrost @jason_trost covert.ioblogposts hlp://www.covert.io/post/18414889381/accumulo- nutch-and-gora hlp://www.covert.io/post/18605091231/accumulo-and- pig 42. QuesNons? Contactme! JoeyEcheverria joey@cloudera.com @fwio Werehiring! 43. 2013Cloudera,Inc.AllRightsReserved. 43

View more