big data as opposed to small data mark whitehorn
TRANSCRIPT
BIG DATA - AS OPPOSED TO SMALL DATA
Mark Whitehorn
2
What is Big data?
Is it really just a marketing campaign?
http://www.perceptualedge.com/articles/visual_business_intelligence/big_data_big_ruse.pdf
“If you’re like me, the mere mention of Big Data now turns your stomach….Why all the fuss? Why, indeed. Essentially, Big Data is a marketing campaign, pure and simple.” Stephen Few
3
Big dataClearly I am not like Stephen Few.
I don’t believe I have a particular axe to grind, I simply find this interesting
This talk is designed to try to explain:• what Big Data is• what characteristics we have found useful• why it may be of interest to you• a paradox
4
Data
All computer applications manipulate data
5
Data
So, in the ’60 and ‘70s we rapidly learnt to separate the data, and its manipulation, from the application
6
Data
So, in the ’60 and ‘70s we rapidly learnt to separate the data, and its manipulation, from the applicationWhich led directly to the development of database engines and, ultimately, relational ones (DB2, Oracle, SQL Server)
7
Data
Data has always existed in two, very broad, flavours…..
• Data that is treated as small, discrete packages and is a good fit with the relational way of storing and querying data
• Data that is not as above
Data is stored in tables
8
Mark Whitehorn
LicenseNo Make Model Year ColourCER 162 C Triumph Spitfire 1965 GreenEF 8972 Bentley Mk. VI 1946 BlackYSK 114 Bentley Mk. VI 1949 Red
Data is stored in tables
9
Mark Whitehorn
LicenseNo Make Model Year ColourCER 162 C Triumph Spitfire 1965 GreenEF 8972 Bentley Mk. VI 1946 BlackYSK 114 Bentley Mk. VI 1949 Red
CarEach table has a name
Data is stored in tables
10
Mark Whitehorn
LicenseNo Make Model Year ColourCER 162 C Triumph Spitfire 1965 GreenEF 8972 Bentley Mk. VI 1946 BlackYSK 114 Bentley Mk. VI 1949 Red
Car
Data isatomic
Data is stored in tables
11
Mark Whitehorn
LicenseNo Make Model Year ColourCER 162 C Triumph Spitfire 1965 GreenEF 8972 Bentley Mk. VI 1946 BlackYSK 114 Bentley Mk. VI 1949 Red
Columns
Car
Data is stored in tables
12
Mark Whitehorn
LicenseNo Make Model Year ColourCER 162 C Triumph Spitfire 1965 GreenEF 8972 Bentley Mk. VI 1946 BlackYSK 114 Bentley Mk. VI 1949 Red
CarColumns
Rows
Data is stored in tables
13
Mark Whitehorn
LicenseNo Make Model Year ColorCER 162 C Triumph Spitfire 1965 GreenEF 8972 Bentley Mk. VI 1946 BlackYSK 114 Bentley Mk. VI 1949 Red
Car
Each row represents a unique entity in the ‘real’ world……
14
15
Data
The manipulation consists typically of sub-setting the data by rows and columns and then doing some sums
16
Data
Note that this kind of manipulation is treating the data as atomic, which is fine, because the relational model assumes atomicity of data
Note also, that the rows are unordered
17
Data
• Data has always existed in two, very broad, flavours…..• Data that is inherently atomic and is a good
fit with the relational way of storing and querying data
• Data that is not as above
Examples
• Examples of ‘other’ data:• Images• Music• Word docs• Sensor data• Web logs• Twitter• Machines
• Point of Sale• Mass spectrometers
18
What’s in a name?
So, what do we call the ‘rest’?• Un-structured?• Semi-structured?• Multi-structured?• Non-relational?• Non-tabular?
19
What’s in a name?
• What about: • Big data?
20
Other definitions?
• V V V v v v v • Volume• Variety• Velocity• Value• Very interesting• Various other words beginning with V…..
21
22
Big Data – not new?
• So why have we focused, for the last 30 years, almost exclusively on the first flavour?
• Because it:• is easy (relatively easy – Jim Gray*)• represents a significant proportion of the
available data
*Jim Gray and Andreas Reuter - Transaction Processing: Concepts and Techniques (1993)Turning Award 1998
23
Big Data has come of age
• Two factors have changed• Rise of the Machines• Increase is computational power
• There is a great synergy here• We are acquiring far more big data and we
have computational power to extract the information it contains
Big Data is hard
• 3 Vs• It is highly variable• We often want to look inside the data
• Frequently non-atomic• Need custom functions for virtually every operation
• find the rotating wing aircraft in the image• Identify the best customer• What does the blog sphere think of our
company?24
• Examples• Log file• Mass spec.• Images
Big Data
25
• Examples• Log file• Mass spectrometer• Image
Big Data
26
• Examples• Log file• Mass spec.• Images
Big Data
27
What is Big Data?• Examples
• Log file• Mass spec.• Images
BIG DATA
Summary so far……
• Just as you can always fit an aircraft engine into a car chassis, you can always put Big Data in a table, but you probably don’t want to
• The analysis is not sub-setting the data by rows and columns
• So each class of big data usually require a (lovingly hand-crafted) custom analysis
30
Case Study
Big Data in the Life Sciences WorldThe massed spectrometers
Why would anyone do that?
31
Human Genome Project$3 billion – 13 Years
Sequencing completed (2003).
32
Human Genome Project
Our genes define us.
Errr…. how does that work exactly?
Human Genome Project$3 billion – 13 Years
33
DNA Protein
blueprint product
What is a protein?
34
Genes contain instructions for creating
proteins
Proteins carry out functions within a cell
GENOME
PROTEOME
Why study proteins
35
Example ProteinsProtein: ACTINFunction: Contracts Muscles
Protein: InsulinFunction: Controls Blood Sugar
O2
Protein: HemoglobinFunction: Carries Oxygen
Protein: KeratinFunction: Forms Hair and Nails
Protein: AntibodyFunction: Fights Viruses
36
20-25,000 genes in the human genome.Every nucleated cell in the same human has the same genome.
But not all genes are active at the same time.Perm any 15-18,00 active proteins in any one cell at any one time.
biS
CIE
NC
E
37
slowly changing millions of years
rapidly changingover a day38
Studying Proteins
Proteins are chopped up using an enzyme to make them easier to measure.
A specialised instrument (Mass Spectrometer) is used to measure (‘weigh’) the small protein fragments.
We can use the mass of the small fragments to carry out intelligent database searches to identify which protein was detected.
39
Protein
MKLNISFPATGCQKLIEVDDERKLRTFYEKRMATEVAADALGEEWKGYVVRISGGNDKQGFPMKQGVLTHGRVRLLLSKGHSCYRPRRTGERKRKSVRGCIVDANLSVLNLVIVKKGEKDIPGLTDTTVPRRLGPKRASRIRKLFNLSKEDDVRQYVVRKPLNKEGKKPRTKAPKIQRLVTPRVLQHKRRRIALKKQRTKKNKEEAAEYAKLLAKRMKEAKEKRQEQIAKRRRLSSLRASTSKSESSQK
Amino Acids
Peptides
40
Mass SpectrometryAn analytical technique for the determination of the elemental composition of a sample.
41
Spectra
P1
P2
P3
42
Mass SpectraFile Sizes: typically several gigabytes per MS run.
Identifications: range from 500-8000 protein identifications.
43
pep TRACKERTRACK. VISUALISE. DISCOVER.
44
80%60%
40%20%
45
Protein Peptide Alignment Map
Normalised Profiles for Synthesis,
Degradation and Turnover
Localisation
Comparison Between Compartments
46
Custom analysis and custom visualisation – vital tools in understanding big data
47
Proteomics Volume 3, Issue 8, Article first published online: 12 AUG 2003
Deisotoping
Base Line Correction Peak Detection
BIOConductor PROcess R Package
Intensive Data Processing Required to derive Information from the raw data
48
“proteomics is much more complicated than genomics . . . while an organism's genome is
more or less constant, the proteome differs from cell to cell
and over time”
Computationally, perhaps three orders of magnitude more
complex than HGP
49
Why bother trying to quantify it?
Because this is payback time.
Documenting the proteome opens the door to a whole new world.
biS
CIE
NC
E
50
So, what is a data scientist?My favourite description comes from Twitter:“Yeah, so I'm actually a data scientist. I just do this barista thing in between gigs.”More cynically:“A data scientist is just an analyst who lives in California.”
biS
CIE
NC
E
51
Possibly more accurate is that a data scientist (DS) is “a better software engineer than any statistician and a better statistician than any software engineer”.
biS
CIE
NC
E
52
DSs are also part artist and part engineer. They need a toolbox of techniques, skills, processes and abilities from which to construct novel solutions. And they need the ability to create a UI that turns their abstract finding into something that the users of the system can understand, so DSs also need the skills to create elegant visualisations that turn raw data into information.
biS
CIE
NC
E
53
And (yes, there’s more) they need to be able to communicate well with people. There is little use in creating a superb analytical process if you can’t communicate how and why it works to the board members.
biS
CIE
NC
E
54
And then there is the curiosity. Duncan Ross (Director of Data Sciences at Teradata) characterised data scientists well:The first and most important trait is curiosity. Insane curiosity. In many walks of life evolution selects against the kind of person who decides to find out what happens “if I push that button”. Data Science selects for it.
biS
CIE
NC
E
55
So, what are the general characteristics of a DS? They include:• insatiable curiosity (see above)• interdisciplinary interests• excellent communication skills • excellent analytical capabilities
biS
CIE
NC
E
56
DSs also need a good working knowledge of:• machine learning techniques• data mining• statistics• maths• algorithm development• code development • data visualisation• multi-dimensional database design and
implementation
biS
CIE
NC
E
57
Specific skills include the technologies to handle big data:• NoSQL databases• Hadoop and related technologies• MapReduce and its implementation on differing
software platforms
biS
CIE
NC
E
58
DSs also have an intimate knowledge of languages such as:• SQL• MDX • R• Functional and OOP languages such as Erlang and
Java
biS
CIE
NC
E
59
Most of all, no matter what they are called, all true data scientists have started playing with some data at 8:00PM and suddenly found it is 3:00AM.
biS
CIE
NC
E
Case Study
TwitterWho loves you?Social/text/sentiment
61
Consider the humble tweet…
62
Consider the humble tweet…
63
As, indeed, Sally Bercow should have done
Consider the humble tweet…
64
As, indeed, Sally Bercow should have done *Innocent Face*
Consider the humble tweet…
I’d just like to apologise for that last slide but I would point out
that it “contained no accusation whatsoever … Mischievous but
not libellous.”
65
Case Study
Oil Rig dataGone fishing
Sensor data
66
Lessons learned
• Engagement
• Choose you battles – look for an area where you can gain competitive advantage
• Choose your platform carefully• Programming – algorithm development• Data scientists
• Custom algorithms • Custom visualisations 67
BIG DATA - AS OPPOSED TO SMALL DATA
60 minutes
Mark Whitehorn