real-time information analysis: social networks and open data

Real-time Information Discovery and Analytics

P L A M E N P E N E VC O - F O U N D E R

KNOWLEDGE GROWSWHERE DATA FLOWS,

Yatrus is building a platform for discovering early

market moving information by using

non-traditional sources for hedge funds and

investments bankers.

www.yatrusanalytics.comData Science Society 18 Jan 2016


Why real-time analysis matters?


Data- big and small, fast and smart, Variety of data

What are the volumes of datathat we are seeing today?

30 BILION PIECES OF CONTENTwere added to face book this past monthby 600 million plus users.

ZYNGA PROCESSES 1 PETABYTE OF CONTENTfor players every day; a volume of data that isunmatched in social game industry.

32 BILLIONS SEARCHESwere perfomed last month.... on Twitter.

MORE THAN 2 BILLION VIDEOSwere watched on YouTube....yesterday.

Will be online, pushing the data created and shared to nearly 8 zettabytes

WORLDWIDE IP TRAFIC will quadruple by 2015

3bilion

A new IDC study says the market

for big technology and services

will grow from 3.2 billion in 2010 to

$16.9 billion in 2015. That’s a 40%

growth CAGR.

$16.9 billion

$3.2 billion

2 /3of surveyed business in North America said big data will become a concern for within the next five years.

business and consumer life

of the data in the world today has been created in the last two years alone.

Everydaycreates 2.5 quintillion

bytes of data per day. 90%

Agriculture

Health Care

Real Estate

Marketing

Cybersecurity

Retail

Energy

Finance

BIG DATA


Context and Semantics

• To put data into context and unite disparate sources

• Semantics seems to be the solution


Context and Semantics

SEMANTIC

WEB

DATAAgent

Knowledge

Graph

Communication

Approach

Urls

Networks

Blog

USER

Dataset

Nodes

Metadata

Interface

Concepts

Class

Content

Algorithm

Detections


Linked Open Data- Knowledge bases and Ontologies

• Knowledge networks

• Dbpedia

• Freebase

• Domain ontologies


Linked Open Data- Knowledge bases and Ontologies

Dbpedia

Riese

US Consus

Data

Worldfact-book

Eurostat

YagoLingvoj

Umbel DBLPHannover

RKBExplorer

LinkedMDB

Flikrwrappr

Revyu

Semanticweb.org

Flikrexporter

RDF Boockmashup

BBCProgrames

Geo-names

Crunchbase

QDOS

Audio-Scrobbler

Music-brainzMyspace

wrapper

BBCPlaycount

DataJamendo

Magna-tune

Wiki-company

Gov-track W3C

WordNet ProjectGuten-

Berg

DBLPBerlin

BBCJohnPeel

Doap-space

FOAFprofiles


Natural language processing and Semantics

• Watson and domain ontolgies and knowledge bases- rea-soning

• Named entity recognition

• Classification

• Watson

Data Science Society 18 Jan 2016

Architectural advances

• Spark

• Lambda Architecture

• Hadoop + graph dbs

• Storm

• Druid

• Cassandra

www.yatrusanalytics.com


Social networks – Twitter “The heartbeat of the world’’

• Twitter specifics

• Value out of twitter data



Social Networks


True fact sayings:

105.779.710Registered USERS

140characters in a message

400milion Active users

39Average age of user

6000Tweets per second

350 000Tweets per minutes

500-700million Tweets per day


Open Data World

• Talking about a variety

• Government data

• Demographics

• Company data

• FInancial


Goverment

Data aggregators

Social data

Weather data Sports data

Google

Markets

Universities and research

News data


Yatrus real-time analytical flow

• Network analysis

• Sentiment analysis

• Complex systems science

• Natural Language Processing

• Machine Learning





Bearish

Bearish

bullish

BullishOptimism Optimism

Capitulation

Despondency

Excitement

Thrill

Euphoria

Anxiety

Denial

Fear

Depresion

Depresion

Hope

ReliefPanic

Complex Systems


• Definitions-Consisting of many diverse and autonomous but interrelated and interdependent components or parts linked through many (dense) intercon-nections.

• Ecosystems, Brains, Societies, the Internet (of Things)

SANDY PENTLAND

“Its all about paying attention to patterns in life and

using that information to help with things like setting

privacy patterns, sharing things with people, notify-

ing people - basically, to help you live your life."





SOCIALPHYSICSby: Alex Pentland


Systems' analytics tools


• Network analysis

• Fitness landscape

• Agent-based modeling


Conclusions


Conclusions


real-time information analysis: social networks and open data

Data & Analytics