sharing a startup’s big data lessons

Sharing a Startup’s Big Data Lessons

Experiences with non-RDBMS solutions at

Who we are

• A search engine• A people

search engine• An influencer

search engine• Subscription-

George Stathis

VP Engineering14+ years of experience building full-stack web software systems with a past focus on e-commerce and publishing. Currently responsible for building engineering capability to enable Traackr's growth goals.

What’s this talk about?

• Share what we know about Big Data/NoSQL:

what’s behind the buzz words?

• Our reasons and method for picking a NoSQL

database

• Share the lessons we learned going through

the process

Big Data/NoSQL: behind the buzz words

What is Big Data?

• 3 Vs:– Volume– Velocity– Variety

What is Big Data? Volume + Velocity

• Data sets too large or coming in at too high a velocity to process using traditional databases or desktop tools. E.g.

big scienceweb logsrfidsensor networkssocial networkssocial datainternet text and documentsinternet search indexingcall detail records

Astronomyatmospheric sciencegenomicsbiogeochemicalmilitary surveillancemedical recordsphotography archivesvideo archiveslarge-scale e-commerce

Traditional static reports

What is Big Data? Variety

• Big Data is varied and unstructured

Analytics, exploration & experimentation

$$$$$$$$

What is Big Data?

• Scaling data processing cost effectively

What is NoSQL?

• NoSQL ≠ No SQL• NoSQL ≈ Not Only SQL• NoSQL addresses RDBMS limitations, it’s not

about the SQL language• RDBMS = static schema• NoSQL = schema flexibility; don’t have to

know exact structure before storing

What is Distributed Computing?

• Sharing the workload: divide a problem into many tasks, each of which can be solved by one or more computers

• Allows computations to be accomplished in acceptable timeframes

• Distributed computation approaches were developed to leverage multiple machines: MapReduce

• With MapReduce, the program goes to the data since the data is too big to move

What is MapReduce?

Source: developer.yahoo.com

What is MapReduce?

• MapReduce = batch processing = analytical• MapReduce ≠ interactive• Therefore many NoSQL solutions don’t

outright replace warehouse solutions, they complement them

• RDBMS is still safe

What is Big Data? Velocity

• In some instances, being able to process large amounts of data in real-time can yield a competitive advantage. E.g.– Online retailers leveraging buying history and click-

though data for real-time recommendations• No time to wait for MapReduce jobs to finish• Solutions: streaming processing (e.g. Twitter

Storm), pre-computing (e.g. aggregate and count analytics as data arrives), quick to read key/value stores (e.g. distributed hashes)

What is Big Data? Data Science

• Emergence of Data Science • Data Scientist ≈ Statistician• Possess scientific discipline & expertise• Formulate and test hypotheses• Understand the math behind the algorithms so

they can tweak when they don’t work• Can distill the results into an easy to understand

story• Help businesses gain actionable insights

Big Data Landscape

Source: capgemini.com

Big Data Landscape

Traackr consumes

Traackr uses these for storage

Traackr adds value in these verticals through our

custom tools

Big Data Landscape

Traackr re-sells

So what’s Traackr and why did we need a NoSQL DB?

Traackr: context

• A cloud computing company as about to launch a new platform; how does it find the most influential IT bloggers on the web that can help bring visibility to the new product? How does it find the opinion leaders, the people that matter?

Traackr: a people search engine

Up to 50 keywords per search!

Traackr: a people search engine

Peopleassearchresults

Contentaggregatedby author

Proprietary 3-scale ranking

Traackr: 30,000 feet

Acquisition Processing Storage & Indexing Services Applications

NoSQL is usually associated with“Web Scale” (Volume & Velocity)

• In terms of users/traffic?

Do we fit the “Web scale” profile?

Source: compete.com

• In terms of the amount of data?

PRIMARY> use traackrswitched to db traackrPRIMARY> db.stats(){

"db" : "traackr","collections" : 12,"objects" : 68226121,"avgObjSize" : 2972.0800625760330,"dataSize" : 202773493971,"storageSize" : 221491429671,"numExtents" : 199,"indexes" : 33,"indexSize" : 27472394891,"fileSize" : 266623699968,"nsSizeMB" : 16,"ok" : 1

That’s a quarter of a terabyte …

Wait! What? My Synology NAS at home can hold 2TB!

No need for us to track the entire web

Web Content

Influencer Content

Not at scale :-)

Variety view of “Web Scale”

Web data is:

Heterogeneous

Unstructured (text)

Source: http://www.opte.org/

Visualization of the Internet, Nov. 23rd 2003

Data sources are

isolated islands of rich

data with lose links to

one another

How do we build a database that models all possible entities found on the web?

Modeling the web: the RDBMS way

Source: socialbutterflyclt.com

{ "realName": "David Chancogne", "title": "CTO", "description": "Web. Geek.\r\nTraackr: http://traackr.com\r\nPropz: http://propz.me", "primaryAffiliation": "Traackr", "email": "dchancogne@traackr.com", "location": "Cambridge, MA, United States", "siteReferences": [ { "siteUrl": "http://twitter.com/dchancogne", "metrics": [ { "value": 216, "name": "twitter_followers_count" }, { "value": 2107, "name": "twitter_statuses_count" } ] }, { "siteUrl": "http://traackr.com/blog/author/david", "metrics": [ { "value": 21, "name": "google_inbound_links" } ] } ]}

Influencer data as JSON

NoSQL = schema flexibility

• In terms of the variety of the data

Traackr’s Datastore Requirements

• Schema flexibility

• Good at storing lots of variable length text

• Batch processing options

Requirement: text storage

Variable text length:

< big variance <140

character tweets

multi-page

blog posts

RDBMS’ answer to variable text length:

Plan ahead for largest value

CLOB/BLOB

Issues with CLOB/BLOG for us:

No clue what largest value is

CLOB/BLOB for tweets = wasted space

NoSQL solutions are great for text:

No length requirements (automated

chunking)

Limited space overhead

Requirement: batch processing

Some NoSQL

solutions come

with MapReduce

Source: http://code.google.com/

Requirement: batch processing

MapReduce + RDBMS:

Possible but proprietary solutions

Usually involves exporting data from

RDBMS into a NoSQL system anyway.

Defeats data locality benefit of MR

A NoSQL option is the right fit

How did we pick a NoSQL DB?

Bewildering number of options (early 2010)

Key/Value Databases• Distributed hashtables• Designed for high load• In-memory or on-disk• Eventually consistent

Column Databases• Spread sheet like• Key is a row id• Attributes are columns• Columns can be grouped

into families

Document Databases• Like Key/Value• Value = Document• Document = JSON/BSON• JSON = Flexible Schema

Graph Databases• Graph Theory G=(E,V)• Great for modeling

networks• Great for graph-based

query algorithms

Bewildering number of options (early 2010)

into families

query algorithms

Trimming optionsKey/Value Databases• Distributed hashtables• Designed for high load• In-memory or on-disk• Eventually consistent

into families

query algorithms

Graph Databases: while we can model our domain as a graph we don’t want to pigeonhole ourselves into this structure.

We’d rather use these tools for specialized data analysis but not as the

main data store.

into families

query algorithms

Memcache: memory-based,we need true persistence

into families

query algorithms

Amazon SimpleDB: not willing to store our data in a proprietary

datastore.

into families

query algorithms

Not willing to store our data in a proprietary datastore.

Redis and LinkedIn’s Project Voldermort: no query filters,

better used as queues or distributed caches

into families

query algorithms

CouchDB: no ad-hoc queries; maturity in early 2010 made us shy away although we did try

early prototypes.

into families

query algorithms

Cassandra: in early 2010, maturity questions, no secondary indexes and no batch processing options

(came later on).

into families

query algorithms

MongoDB: in early 2010, maturity questions, adoption questions

and no batch processing options.

into families

query algorithms

Riak: very close but in early 2010, we had adoption questions.

into families

query algorithms

HBase: came across as the most mature at the time, with several deployments, a

healthy community, "out-of-the box" secondary indexes through a contrib and

support for batch processing using Hadoop/MR .

Lessons Learned

Challenges

- Complexity

- Missing Features

- Problem solution fit

- Resources

Rewards

- Choices

- Empowering

- Community

- Cost

Rewards: ChoicesKey/Value Databases• Distributed hashtables• Designed for high load• In-memory or on-disk• Eventually consistent

into families

query algorithms

Rewards: Choices

Lessons Learned

Challenges

- Complexity

- Missing Features

- Resources

Rewards

- Choices

- Empowering

- Community

- Cost

When Big-Data = Big Architectures

Source: http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html

Must have a Hadoop HDFS cluster of at least 2x replication

factor nodes

Must have an odd number of

Zookeeper quorum nodes

Then you can run your Hbase nodes but it’s recommended to

co-locate regionservers with hadoop datanodes so you have

to manage resources.

Master/slave architecture means a single point of failure,

so you need to protect your master.

And then we also have to manage the MapReduce

processes and resources in the Hadoop layer.

Jokes aside, no one said open source was easy to use

To be expected

• Hadoop/Hbase are

designed to move

mountains

• If you want to move big

stuff, be prepared to

sometimes use big

equipment

What it means to a startup

Development capacity before

Development capacity after

Congrats, you are now a sysadmin…

Lessons Learned

Challenges

- Complexity

- Missing Features

- Resources

Rewards

- Choices

- Empowering

- Community

- Cost

Mapping an saved search to a column store

Ranks References to influencer records

Unique key

“attributes” column family

for general attributes

“influencerId” column familyfor influencer ranks and foreign keys

“name” attribute

Influencer ranks can be attribute names as well

Can get pretty long so needs indexing and pagination

Problem: no out-of-the-box row-based indexing and pagination

Jumping right into the code

Lessons Learned

Challenges

- Complexity

- Missing Features

- Resources

Rewards

- Choices

- Empowering

- Community

- Cost

a few months later…

Need to upgrade to Hbase 0.90

• Making sure to remain on recent code base

• Performance improvements

• Mostly to get the latest bug fixes

No thanks!

Looks like something is missing

Our DB indexes depend on this!

Let’s get this straight

• Hbase no longer comes with secondary

indexing out-of-the-box

• It’s been moved out of the trunk to GitHub

• Where only one other company besides us

seems to care about it

Only one other maintainer besides us

Development capacity

Congrats, you are now an hbase

contrib maintainer…

Lessons Learned

Challenges

- Complexity

- Missing Features

- Resources

Rewards

- Choices

- Empowering

- Community

- Cost

Homegrown Hbase Indexes

Rows have id prefixes that can be efficiently scanned using STARTROW and STOPROW filters

Row ids for Posts

Find posts for influencer_id_1234

Row ids for Posts

Find posts for influencer_id_5678

Row ids for Posts

• No longer depending on

unmaintained code

• Work with out-of-the-box Hbase

installation

You are back but you still need to

maintain indexing logic

a few months later…

Cracks in the data modelhuffingtonpost.com

huffingtonpost.com

http://www.huffingtonpost.com/arianna-huffington/post_1.htmlhttp://www.huffingtonpost.com/arianna-huffington/post_2.html

http://www.huffingtonpost.com/arianna-huffington/post_3.html

http://www.huffingtonpost.com/shaun-donovan/post1.htmlhttp://www.huffingtonpost.com/shaun-donovan/post2.html

http://www.huffingtonpost.com/shaun-donovan/post3.html

writes for

authored by

published under

writes for

authored by

published under

huffingtonpost.com

writes for

authored by

published under

writes for

authored by

published under

Denormalized/duplicated for fast runtime access

and storage of influencer-to-site relationship

properties

huffingtonpost.com

writes for

authored by

published under

writes for

authored by

published under

Content attribution logic could sometimes mis-attribute posts because of the

duplicated data.

huffingtonpost.com

writes for

authored by

published under

writes for

authored by

published under

Exacerbated when we started tracking people’s content on a daily basis in mid-

Fixing the cracks in the data model

huffingtonpost.com

writes for

authored by

published under

writes for

authored by

published under

Normalize the sites

Fixing the cracks in the data model

• Normalization requires stronger

secondary indexing

• Our application layer indexing would

need revisiting…again!

Psych! You are back to writing indexing

Lessons Learned

Challenges

- Complexity

- Missing Features

- Resources

Rewards

- Choices

- Empowering

- Community

- Cost

Traackr’s Datastore Requirements (Revisited)

• Out-of-the-box SECONDARY INDEX support!

• Simple to use and administer

NoSQL picking – Round 2 (mid 2011)

into families

query algorithms

into families

query algorithms

into families

query algorithms

Graph Databases: we looked at Neo4J a bit closer but passed again

for the same reasons as before.

into families

query algorithms

Memcache: still no

into families

query algorithms

Amazon SimpleDB: still no.

into families

query algorithms

Not willing to store our data in a proprietary datastore.

Redis and LinkedIn’s Project Voldermort: still no

into families

query algorithms

CouchDB: more mature but still no ad-hoc queries.

into families

query algorithms

Cassandra: matured quite a bit, added secondary indexes and batch processing

options but more restrictive in its’ use than other solutions. After the Hbase lesson,

simplicity of use was now more important.

into families

query algorithms

Riak: strong contender still but adoption questions remained.

into families

query algorithms

MongoDB: matured by leaps and bounds, increased adoption, support from 10gen, advanced indexing out-of-the-box as well as some batch processing

options, breeze to use, well documented and fit into our existing code base very nicely.

Lessons Learned

Challenges

- Complexity

- Missing Features

- Resources

Rewards

- Choices

- Empowering

- Community

- Cost

Immediate Benefits

• No more maintaining custom application-layer

secondary indexing code

Yay! I’m back!

Immediate Benefits

• Single binary installation greatly simplifies

administration

Honestly, I thought I’d never see you

guys again!

Immediate Benefits

• Single binary installation greatly simplifies

administration

• Our NoSQL could now support our domain

many-to-many relationship

Modeling an influencer

Embedded list of references to sites augmented with

influencer-specific site attributes (e.g.

percent contribution to content)

{ ”_id": "770cf5c54492344ad5e45fb791ae5d52”, "realName": "David Chancogne", "title": "CTO", "description": "Web. Geek.\r\nTraackr: http://traackr.com\r\nPropz: http://propz.me", "primaryAffiliation": "Traackr", "email": "dchancogne@traackr.com", "location": "Cambridge, MA, United States", "siteReferences": [ { "siteId": "b31236da306270dc2b5db34e943af88d", "contribution": 0.25 }, { "siteId": "602dc370945d3b3480fff4f2a541227c", "contribution": 1.0 } ]}

Modeling an influencer

siteId indexed for “find influencers

connected to site X”

> db.influencers.ensureIndex({siteReferences.siteId: 1});> db.influencers.find({siteReferences.siteId: "602dc370945d3b3480fff4f2a541227c"});

{ ”_id": "770cf5c54492344ad5e45fb791ae5d52”, "realName": "David Chancogne", "title": "CTO", "description": "Web. Geek.\r\nTraackr: http://traackr.com\r\nPropz: http://propz.me", "primaryAffiliation": "Traackr", "email": "dchancogne@traackr.com", "location": "Cambridge, MA, United States", "siteReferences": [ { "siteId": "b31236da306270dc2b5db34e943af88d", "contribution": 0.25 }, { "siteId": "602dc370945d3b3480fff4f2a541227c", "contribution": 1.0 } ]}

Other Benefits

• Ad hoc queries and reports became easier to write with

JavaScript: no need for a Java developer to write map reduce code

to extract the data in a usable form like it was needed with Hbase.

• Simpler backups: Hbase mostly relied on HDFS redundancy; intra-

cluster replication is available but experimental and a lot more

involved to setup.

• Great documentation

• Great adoption and community

looks like we found the right fit!

We have more of this

And less of this

Recap & Final Thoughts

• 3 Vs of Big Data:– Volume– Velocity– Variety Traackr

• Big Data technologies are complementary to SQL and RDBMS

• Until machines can think for themselves Data Science will be increasingly important

Recap & Final Thoughts

• Be prepared to deal with less mature tech

• Be as flexible as the data => fearless refactoring

• Importance of ease of use and administration

cannot be overstated for a small startup

sharing a startup’s big data lessons

fast runtime

telecommunication

maintaining

traackrs datastore

large hadron

based query

batch processing

existing code

Documents

lessons learned: maximizing the potential of work-sharing...

local conservation networks – learning lessons, sharing

social forestry & the paris agreement: lessons for benefit...

lessons for redd+ benefit-sharing mechanisms from anti

sharing lessons - gec

sharing lessons on mangrove...

sharing lessons and perspectives in marketing

hebrew bible as data: laboratory, sharing, lessons

lessons learned from asian tsunami disaster: sharing...

deepening understanding and pws sharing lessons with...

the need to know: information sharing lessons for disaster

the startup’s guide to marketing and...

equitable redd+ benefit sharing? lessons from brazil and...

knowledge sharing in development agencies: lessons from...

benefit sharing and sustainable hydropower: lessons...

sharing lessons learned on structural safety - fe… ·...

a theoretical analysis of the lean startup’s product

recording and sharing lessons with video

sharing lessons of smallholders’ pig system in south asia

lessons learned from ga4gh - human frontier science ......