Upload File

meshwork - insight data engineering project

9
RANKING THE INTERNETZ WITH MESHWORK Justin Cano Insight Data Engineering Fellow

Upload: justin-cano

Post on 14-Aug-2015

52 views

Category:

Technology


1 download

Report

Tags:

TRANSCRIPT

Page 1: Meshwork - Insight Data Engineering Project

RANKING THE INTERNETZ WITH MESHWORK Justin Cano Insight Data Engineering Fellow

Page 2: Meshwork - Insight Data Engineering Project

Motivation • The internet is huge • How does your page rank amongst others in your mesh?

• What is the reach of your website? Which pages are affecting your page rank?

Data Source • Common Crawl Organization

• More than 7 years of web page data, over 500TB • CC April 2015 web corpus ~168TB • Processed ~445GB for project • Readily available in S3

Page 3: Meshwork - Insight Data Engineering Project

Meshwork – your mesh in a network http://www.jcano.me/meshwork

Page 4: Meshwork - Insight Data Engineering Project

Pipeline

Data from S3 (source of truth)

REST

Page 5: Meshwork - Insight Data Engineering Project

Data

Raw (WARC format) Extraction

Edge List

Page 6: Meshwork - Insight Data Engineering Project

… …

Data Flow

…

…

Link edge data (vertexId, pageRank)

Page 7: Meshwork - Insight Data Engineering Project

Scaling up Page Rank job… spark-submit --class pageRank...4 nodes

=> 20 hours >3 mil records

6 nodes

=> 16 hours

8 nodes

=> 10 hours

Page 8: Meshwork - Insight Data Engineering Project

About Me Justin Cano UC Riverside BS Computer Engineering

Previous work experience Software Engineer @

Hobbies I like building things!

•  Hardware, software Learning and using new technologies Moviegoer Outdoor activities: biking, snowboarding Interests: design, app dev Favorite TV Shows: Futurama & The Daily Show

Embedded Systems Developer @

Software Engineer Intern @

Page 9: Meshwork - Insight Data Engineering Project

TGF-b2-Induced Invadosomes in Human Trabecular Meshwork Cells

PREMIEREISSUE ADVANTAGE - mesco.com.pl · Stent Analysis Expand Students’ Exposure to Biomedical Engineering Engineering students gain insight into the physics of medical devices

ORIGINAL ARTICLE Trabecular meshwork failure in a model …d-scholarship.pitt.edu/30990/1/Trabecular meshwork failure in a... · ORIGINAL ARTICLE Trabecular meshwork failure in a

Dmu insight magazine engineering and sustainable development

Effects of Mechanical Stress on Human Trabecular Meshwork

Insight Data Engineering Project: Rettiwt

Freeze-thaw decellularization of the trabecular meshwork in an … · 2017. 2. 5. · Freeze-thaw decellularization of the trabecular meshwork in an ex vivo eye perfusion model Alternatives:

Case Study Meshwork 2020 Climate Leadership Campaign Final

INSIGHT ENGINEERING SYSTEMS PVT. LTD. · Insight Engineering Systems improve its sales and marketing function. The software has helped the company enhance the quality of its proposals

2D- and 3D-cultures of human trabecular meshwork cells: A preliminary assessment … · 2019-09-10 · RESEARCH ARTICLE 2D- and 3D-cultures of human trabecular meshwork cells: A preliminary

Impact of Pigment Dispersion on Trabecular Meshwork Cellsd-scholarship.pitt.edu/33741/1/Chao Wang Impact of... · 2018. 1. 26. · Impact of Pigment Dispersion on Trabecular Meshwork

An Insight into Chemical Engineering m subbu

Enterprise Vault Overview - Coming Computer Engineering · 2017-12-21 · Better File Governance with Data Insight and Enterprise Vault •Integrates Data Insight with File System

Meshwork - Insight Data Engineering Project

Case Study Meshwork 2020 Climate Leadership Campaign 1.5

SUMMER 2010 ISSUE 01 Engineering INSIGHT...2 engineering insight [ Summer 2010 - Issue 01] INTRODUCTION Welcome from the Dean of Engineering I would like to welcome you to the first

Insight Data Engineering - Week 4 DRAFT

Insight Innovation Challenge: Innovation Engineering – Fail Fast, Fail Cheap, Succeed Faster by Adam Hansen of Ideas To Go - Presented at the Insight Innovation eXchange North America

Human and monkey trabecular meshwork accumulate …iovs.arvojournals.org/pdfaccess.ashx?url=/data/journals/iovs/... · tutively in human and monkey trabecular meshwork ... crystallin

ESTABLISHING CANINE TRABECULAR MESHWORK CELL … · ESTABLISHING CANINE TRABECULAR MESHWORK CELL CULTURE By Hsiang-Rong Tsai Glaucoma is a leading cause of incurable blindness. The

resource.thaihealth.or.thresource.thaihealth.or.th/system/files/documents/1_201.pdf2. Phacolytic glaucoma Liu trabeccutar meshwork 3. Phacoanaphylactic endophthalmitis Lau{unn antigen

Automotive Insight - UMTRI...Automotive Insight: About Us Skilled. Trusted. Proven. Development 3 Manufacturing Engineering New Technologies JVs Lightweighting M&A Strategic Consulting

ANALYST INSIGHT Model-based Systems Engineering: An ... · apply a fully integrated systems engineering approach that covers data management, modeling, simulation, and validation

The Art of Insight in Science and Engineering: …51:35UTC/rev78ca0ee9dfae The Art of Insight in Science and Engineering Mastering Complexity Sanjoy Mahajan The MIT Press Cambridge,

An Insight into Chemical Engineering - msubbumsubbu.in/book/insight.pdf · An Insight into Chemical Engineering Through Instant Notes, Objective Types and Problems (Useful for GATE

Prarthana Insight Data Engineering Project

Method and Insight from the Experts Revolutionizing Mechanical Engineering

For Fun and Insight Leon King Computer Engineering Technology Network Support Technician

An Insight into Chemical Engineering

TrackademiX Insight Data Engineering Project

Sharing Knowledge - REAAA...• Futureintech Programme • Promote engineering • Insight into the industry • Presenting to school students • Career in engineering • Challenges

Chris Qualls Director, Engineering & Technology Boehringer-Ingelheim: Insight into Advanced Manufacturing

ENGINEERS INSIGHT · 2017-11-29 · Engineers Insight Communication Skills Being an engineering student, you should take every available opportunity to practise your communication

Lecture 28: Mathematical Insight and Engineering

Engineering into Social Deep Insight