hadoop for microsoft devssddconf.com/.../hadoop_kickstarter_for_microsoft_devs.pdf ·...
TRANSCRIPT
![Page 1: Hadoop For Microsoft Devssddconf.com/.../Hadoop_KickStarter_For_Microsoft_Devs.pdf · 2015-05-05 · Storing & Querying Big Data in Hadoop Distributed File System ( HDFS ) Unstructured](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f32c2cd21b03e1dd079d577/html5/thumbnails/1.jpg)
Hadoop Kickstarter For Microsoft Devs
By Gary Short
Duncodin Limited
www.duncodin.it
![Page 2: Hadoop For Microsoft Devssddconf.com/.../Hadoop_KickStarter_For_Microsoft_Devs.pdf · 2015-05-05 · Storing & Querying Big Data in Hadoop Distributed File System ( HDFS ) Unstructured](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f32c2cd21b03e1dd079d577/html5/thumbnails/2.jpg)
Introduction
• Gary Short• Microsoft MVP C#• Freelance data scientist• Big Data / architect / engineer• HDInsight / Hadoop / Pig / Hive• Predictive Analytics• Machine Vision• Computational Linguistics• [email protected]• @garyshort
Image © @Blackmarble
![Page 3: Hadoop For Microsoft Devssddconf.com/.../Hadoop_KickStarter_For_Microsoft_Devs.pdf · 2015-05-05 · Storing & Querying Big Data in Hadoop Distributed File System ( HDFS ) Unstructured](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f32c2cd21b03e1dd079d577/html5/thumbnails/3.jpg)
Agenda
• What problem does Hadoop solve?
• How do I install it?
• How do I get my C# code running on it?
• Questions?
![Page 4: Hadoop For Microsoft Devssddconf.com/.../Hadoop_KickStarter_For_Microsoft_Devs.pdf · 2015-05-05 · Storing & Querying Big Data in Hadoop Distributed File System ( HDFS ) Unstructured](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f32c2cd21b03e1dd079d577/html5/thumbnails/4.jpg)
Demo – What Problem Does Hadoop Solve?
![Page 5: Hadoop For Microsoft Devssddconf.com/.../Hadoop_KickStarter_For_Microsoft_Devs.pdf · 2015-05-05 · Storing & Querying Big Data in Hadoop Distributed File System ( HDFS ) Unstructured](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f32c2cd21b03e1dd079d577/html5/thumbnails/5.jpg)
You Just Swapped One Set of Problems For Another!
![Page 6: Hadoop For Microsoft Devssddconf.com/.../Hadoop_KickStarter_For_Microsoft_Devs.pdf · 2015-05-05 · Storing & Querying Big Data in Hadoop Distributed File System ( HDFS ) Unstructured](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f32c2cd21b03e1dd079d577/html5/thumbnails/6.jpg)
Hadoop Architecture – Data Storage
![Page 7: Hadoop For Microsoft Devssddconf.com/.../Hadoop_KickStarter_For_Microsoft_Devs.pdf · 2015-05-05 · Storing & Querying Big Data in Hadoop Distributed File System ( HDFS ) Unstructured](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f32c2cd21b03e1dd079d577/html5/thumbnails/7.jpg)
Hadoop Architecture – Map Reduce
![Page 8: Hadoop For Microsoft Devssddconf.com/.../Hadoop_KickStarter_For_Microsoft_Devs.pdf · 2015-05-05 · Storing & Querying Big Data in Hadoop Distributed File System ( HDFS ) Unstructured](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f32c2cd21b03e1dd079d577/html5/thumbnails/8.jpg)
How do I Install It?
![Page 9: Hadoop For Microsoft Devssddconf.com/.../Hadoop_KickStarter_For_Microsoft_Devs.pdf · 2015-05-05 · Storing & Querying Big Data in Hadoop Distributed File System ( HDFS ) Unstructured](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f32c2cd21b03e1dd079d577/html5/thumbnails/9.jpg)
![Page 10: Hadoop For Microsoft Devssddconf.com/.../Hadoop_KickStarter_For_Microsoft_Devs.pdf · 2015-05-05 · Storing & Querying Big Data in Hadoop Distributed File System ( HDFS ) Unstructured](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f32c2cd21b03e1dd079d577/html5/thumbnails/10.jpg)
![Page 11: Hadoop For Microsoft Devssddconf.com/.../Hadoop_KickStarter_For_Microsoft_Devs.pdf · 2015-05-05 · Storing & Querying Big Data in Hadoop Distributed File System ( HDFS ) Unstructured](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f32c2cd21b03e1dd079d577/html5/thumbnails/11.jpg)
![Page 12: Hadoop For Microsoft Devssddconf.com/.../Hadoop_KickStarter_For_Microsoft_Devs.pdf · 2015-05-05 · Storing & Querying Big Data in Hadoop Distributed File System ( HDFS ) Unstructured](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f32c2cd21b03e1dd079d577/html5/thumbnails/12.jpg)
![Page 13: Hadoop For Microsoft Devssddconf.com/.../Hadoop_KickStarter_For_Microsoft_Devs.pdf · 2015-05-05 · Storing & Querying Big Data in Hadoop Distributed File System ( HDFS ) Unstructured](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f32c2cd21b03e1dd079d577/html5/thumbnails/13.jpg)
![Page 14: Hadoop For Microsoft Devssddconf.com/.../Hadoop_KickStarter_For_Microsoft_Devs.pdf · 2015-05-05 · Storing & Querying Big Data in Hadoop Distributed File System ( HDFS ) Unstructured](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f32c2cd21b03e1dd079d577/html5/thumbnails/14.jpg)
![Page 15: Hadoop For Microsoft Devssddconf.com/.../Hadoop_KickStarter_For_Microsoft_Devs.pdf · 2015-05-05 · Storing & Querying Big Data in Hadoop Distributed File System ( HDFS ) Unstructured](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f32c2cd21b03e1dd079d577/html5/thumbnails/15.jpg)
How do I get my C# Code Running?
![Page 16: Hadoop For Microsoft Devssddconf.com/.../Hadoop_KickStarter_For_Microsoft_Devs.pdf · 2015-05-05 · Storing & Querying Big Data in Hadoop Distributed File System ( HDFS ) Unstructured](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f32c2cd21b03e1dd079d577/html5/thumbnails/16.jpg)
Say “word count” one more time!
![Page 17: Hadoop For Microsoft Devssddconf.com/.../Hadoop_KickStarter_For_Microsoft_Devs.pdf · 2015-05-05 · Storing & Querying Big Data in Hadoop Distributed File System ( HDFS ) Unstructured](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f32c2cd21b03e1dd079d577/html5/thumbnails/17.jpg)
Which one will win?
![Page 18: Hadoop For Microsoft Devssddconf.com/.../Hadoop_KickStarter_For_Microsoft_Devs.pdf · 2015-05-05 · Storing & Querying Big Data in Hadoop Distributed File System ( HDFS ) Unstructured](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f32c2cd21b03e1dd079d577/html5/thumbnails/18.jpg)
Demo - Streaming
![Page 19: Hadoop For Microsoft Devssddconf.com/.../Hadoop_KickStarter_For_Microsoft_Devs.pdf · 2015-05-05 · Storing & Querying Big Data in Hadoop Distributed File System ( HDFS ) Unstructured](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f32c2cd21b03e1dd079d577/html5/thumbnails/19.jpg)
I’m Not Gonna Lie, That Was a Ballache.Is there an easier way?
![Page 20: Hadoop For Microsoft Devssddconf.com/.../Hadoop_KickStarter_For_Microsoft_Devs.pdf · 2015-05-05 · Storing & Querying Big Data in Hadoop Distributed File System ( HDFS ) Unstructured](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f32c2cd21b03e1dd079d577/html5/thumbnails/20.jpg)
Demo - SDK
![Page 21: Hadoop For Microsoft Devssddconf.com/.../Hadoop_KickStarter_For_Microsoft_Devs.pdf · 2015-05-05 · Storing & Querying Big Data in Hadoop Distributed File System ( HDFS ) Unstructured](https://reader034.vdocuments.net/reader034/viewer/2022042417/5f32c2cd21b03e1dd079d577/html5/thumbnails/21.jpg)
Questions?
• Gary Short
• Duncodin Limited
• Freelance data scientist
• Big Data architect / engineer
• www.duncodin.it
• @garyshort
Image © @Blackmarble