data defined discipline (ddd)
TRANSCRIPT
JianQiao Data Talk 2014 Use by Permission 1
Data Defined Discipline (DDD)
- Data Exploration and Data Security
Zhixiong Chen, PhD, CISSP
Dept. of MATH & CS, Mercy College
JianQiao Data Talk 2014 Use by Permission 2
Data Within the Internet
F: 41000 posts/sec, 1.8M likes/sec, 350GB/sec3.5billion/day, 1,5trillion/sec, 30PetaByte/day
截至2012年12月底，中国网民规模达 5.64亿中国手机网民规模为4.20亿
JianQiao Data Talk 2014 Use by Permission 3
Data Beyond of Internet
500Million+ Connected Devices
Printers WebCam
Routers
Traffic Lights
Traffic Sensors
Connected SCADA Systems
Earth Monitors
Ocean Monitors
Home Automation
Billions+ of SensorsGPS
JianQiao Data Talk 2014 Use by Permission 4
• Volume, Velocity, Variety
• Veracity (Noise), Validity, Volatility
• Privacy, Security
Insights - BIG
Big Opportunity
JianQiao Data Talk 2014 Use by Permission 5
“Curse” Project – Rated R
Objectives
• Which states curse the most
• What users curse the most
• Which states curse the least
• What users curse the most
• US curse map
• What cursing words are most
popular
Data Source• Twitter API
• Public Stream
• Geo-location Enabled
• Cursing parse
Instances• ~500,000 messages (in
10 days)
• ~850,000 curses
• ~175,000 users
Wayne Campbell & Mihail Russu
JianQiao Data Talk 2014 Use by Permission 6
JianQiao Data Talk 2014 Use by Permission 7
Top tenFuck/shit/ass/fucking/bitch/damn/nigga/niggas/bitches/lmfao/
Laughing my fucking ass (arse) off
JianQiao Data Talk 2014 Use by Permission 8
Wi-Fi Access Points– Rated PG13
to see if data can be trained to identify the physical location of a device without GPS coordinates
Brandon Kearns and Matthew Murfitt
JianQiao Data Talk 2014 Use by Permission 9
Bio-Weapon Potential– Rated T
identifying the various bioweapon design targets for a disease of interest through the text mining of the biomedical literature
Christopher M. Frenz & Samer AlMazahreh
bioweapons potential has dramatically changed with advances in genomics and other aspects of biotechnologyDesigner bioweapons are now a reality.
JianQiao Data Talk 2014 Use by Permission 10
Unigram # Occurrences Bigram # Occurrences2 Trigram # Occurrences3
toxin 1401 toxin components 40 Anthrax lethal toxin 44
vaccine 1117 antibiotic prophylaxis 39 B anthracis infection 44
response 1020 immunoglobulin G 35 virulence factor of 43
virulence 996 lethal challenge 34 of anthrax vaccine 43
protective 888 vaccine efficacy 23 the immune response 43
lethal 884 emergency response 18 and treatment of 43
bioterrorism 878 animal model 18 anthrax lethal factor 42
antigen 833 disease outbreaks 18 recombinant protective antigen 41
disease 749 tumor necrosis 14 innate immune response 37
spores 720 health emergencies 14 morbidity and mortality 36
immune 701 respiratory syndrome 14 anti PA IgG 36
infection 690 anti spore 14 the pathogenesis of 36
bioterror 629 health emergency 14 the virulence of 36
factor 580 toxin production 14 threat of bioterrorism 35
agents 569 mutant strains 14 cases of anthrax 35
JianQiao Data Talk 2014 Use by Permission 11
Obesity in Face-to-Face Network
The Spread of Obesity in a Large Social Network Over 32 Years by N.A. Christakis and J.H. Fowler
JianQiao Data Talk 2014 Use by Permission 12
Political blogs
Data from the blogosphere. Shown is a link structure within a community of political blogs (from 2004),where red nodes indicate conservative blogs, and blue liberal. Orange links go from liberal to conservative, and purple ones from conservative to liberal. The size of each blog reflects the number of other blogs that link to it.
JianQiao Data Talk 2014 Use by Permission 13
• Google flu trends
• Social Computing– Social informatics
– Social Intelligence
• Picturing Science
• eco-network (US launched in 2011)
Social-based Applications
JianQiao Data Talk 2014 Use by Permission 14
More…
JianQiao Data Talk 2014 Use by Permission 15
• Thousand years ago: science was empirical describing natural phenomena
• Last few hundred years: theoretical branch using models, generalizations– Art
– Philosophy
– Science
– Medicine
– biology
– Engineering
Science Paradigms - Jim Gray
JianQiao Data Talk 2014 Use by Permission 16
• Last few decades: a computational branch simulating complex phenomena
• Today: data exploration unify theory, experiment, and simulation – Data captured by instruments or generated by
simulator– Processed by software– Information/Knowledge stored in computer– Scientist analyzes database / files using data
management and statistics
Science Paradigms
JianQiao Data Talk 2014 Use by Permission 17
From Jim Gray
Experiments &Instruments
Simulations
answers
questions
Literature
Other Archives facts
facts?
JianQiao Data Talk 2014 Use by Permission 18
Disciplines that are built upon relevant data. It includes data collection, storage, cleansing, understanding, visualization, provenance, intelligence
Data Defined Disciplines
JianQiao Data Talk 2014 Use by Permission 19
• In every perspective of the human life– Myspace
– Instgram
– More in the tables listed
• Landscape
pervasive
JianQiao Data Talk 2014 Use by Permission 20
• Also known as information markets, decision markets, idea futures, event derivatives, or virtual markets
• Make predictions on the outcome of hundreds of real-world events
• Find the probability of a predefined, uncertain future event from market prices
• Get correct outcome (honest) than desired outcome (poll)– Anonymous ve Real Name
Prediction Market – data driven
JianQiao Data Talk 2014 Use by Permission 21
• the University of Iowa Tippie College of Business
• buy and sell contracts based on political election results and economic indicators
• academic traders.
Iowa Electronic Markets
JianQiao Data Talk 2014 Use by Permission 22
• a platform where you make predictions by buying and selling shares
• An exchange
• YES/NO proposition
• Settlement date: yes-$10, no-$0– Vs Future market
Intrade
JianQiao Data Talk 2014 Use by Permission 23
Hollywood Stock Exchange
• a web-based, multiplayer game
• simulated money to buy and sell "shares" of actors, directors, upcoming films, and film-related options
JianQiao Data Talk 2014 Use by Permission 24
• IOWA IEM• Univ Virginia Vecon Lab• MIT Center for Collective Intelligence• Pennsylvania State University ,Laboratory for
Economics Management and Auctions, • Iowa Health Prediction Market• CNN Political Market• Financial Times, FT PRED$CT• WSJ Political Market
More Research Oriented Examples
JianQiao Data Talk 2014 Use by Permission 25
• Betfair
• Foresight Exchange
• Inkling Markets
• TraderSports
• NewsFutures
• Darpa project on next events – cancelled by Congress
• DoD missile defense projects prediction missile test
MorePrediction / bet / gambling
JianQiao Data Talk 2014 Use by Permission 26
Big Challenging
JianQiao Data Talk 2014 Use by Permission 27
Not So Anonymous Project
Sharing Medical InfoHealthcare Data AnalysisMedical Data AnalysisPrescription Data AnalysisDICOM Data
“Under federal privacy law, this data sharing is perfectly legal. As a safeguard, part of the HIPAA requires that a person “with appropriate knowledge of and experience with generally accepted statistical and scientific principles and methods” must certify that there is a “very small” risk of re-identification by an “anticipated recipient” of the data.”
JianQiao Data Talk 2014 Use by Permission 28
DatabaseData Mart
Data WarehouseData Mining
Machine LearningVisualization
autonomic computing
Analytics
IBM NetezzaOracle Exadata
MS IMDBMongoDB
HbaseCassandraCouchDB
DBA
Social InternetMapReduce
Big Table Google File Sys
Hadoop Dist File SysHbase/Hadoop
CloudDistributed
ServicesOn-demandingInfrastructure
JianQiao Data Talk 2014 Use by Permission 29
Q and A