[2b4]live broadcasting 추천시스템
TRANSCRIPT
-
Live Broadcasting
| 2014.9.30
-
()
-
() TV
-
() TV
-
() TV .
-
.
-
TV?
BJ
-
TV?
SNS
-
TV0255075100
100 K 3 M 20 M 20 M 120
-
TV
Touch the void
?
AB
-
?
-
!
-
!
-
!
-
!
-
!Watcha
http://watch.net
-
! .
-
?
-
Touching the void
: http://www.elhablador.com/blog/wp-content/uploads/2012/05/touching-the-void.jpg
http://www.elhablador.com/blog/wp-content/uploads/2012/05/touching-the-void.jpg
-
1988
!
Touching the void
!!!
-
10 Into Thin Air !!!
!
Touching the
void
: www.amazon.com
http://www.amazon.com
-
0
25
50
75
100
-
0
250
500
750
1000
-
!!
-
57% ! !
.
-
?
-
. !
!
! , !
.
-
! -
-
! -
-
TV !
-
TV .
-
BJ BJ `
BJ
BJ
-
BJ
-
1.0 () 0.0
[0.0 , 1.0]
-
0.0
[0.0 , 1.0]
uno !
BJID
10001 rani0224 10 9
10001 afkbo1 120 95
10001 afmlb1 32 93
uno !
BJID Y/N
10001 rani0224 T
10001 afkbo1 T
uno !
BJID
10001 rani0224 50
10001 afkbo1 10
uno !
BJID
10001 rani0224 0.57
10001 afkbo1 0.73
10001 afmlb1 0.21
uno !
BJID Y/N
10001 rani0224 1.0
10001 afkbo1 1.0
uno !
BJID
10001 rani0224 0.5
10001 afkbo1 0.1
Boolean Value !
[ F, T ] [ 0 , 1]
Scale !
[0 , 100] [0.0 1.0]
1.0 ()
-
0.0
[0.0 , 1.0]
0
7.5
15
22.5
30
D D -2 D -4 D -6 D -8 D -10 D -12 D -14 D -16 D -18 D -20 D -22 D -24 D -26 D -28
1.0 ()
-
0.0
[0.0 , 1.0]
0
7.5
15
22.5
30
D D -2 D -4 D -6 D -8 D -10 D -12 D -14 D -16 D -18 D -20 D -22 D -24 D -26 D -28
1.0 ()
[ Tips. ] = () = D/1 + D/2 + D/3 D/30 = ( () = ( log() )
-
0.0
[0.0 , 1.0]
0
0.125
0.25
0.375
0.5
0.1
0.20.2
0.5
= 1.0
1.0 ()
-
0.0
0.623573 0.57521 0.34565 0.21736
1.0 ()
?
-
0.0
0.623573 0.57521 0.34565 0.21736
1.0 ()
HDFS HQL
Redis Cluster
Sorted Set
Hive External Table
HQL
BJ HQL ( )
Redis Save (UDF) HQL
HQL
HQL
HQL
HQL
-
0.0
0.623573 0.57521 0.34565 0.21736
1.0 ()
HDFS HQL
Redis Cluster
Sorted Set
HIVE External Table
HQL
Redis Save (UDF) HQL
HQL
HQL
HQL
HQL
MR-Hive
Compatible
BJ HQL ( )
-
0.0
0.623573 0.57521 0.34565 0.21736
1.0 ()
HDFS HQL
Redis Cluster
Sorted Set
HIVE External Table
HQL
Merge HQL
Redis Save (UDF) HQL
HQL
HQL
HQL
HQL
MR-Hive
Compatible
Query Base Processing
-
0.0
0.623573 0.57521 0.34565 0.21736
1.0 ()
HDFS HQL
Redis Cluster
Sorted Set
HIVE External Table
HQL
Merge HQL
Redis Save (UDF) HQL
HQL
HQL
HQL
HQL
MR-Hive
Compatible
Query Base ProcessingIn
Memory Store
-
BJ
-
1.0
0.0
Clustering BJ Matrix
BJ
BJ
-
1.0
0.0
Clustering BJ Matrix
n=bj, m=cluster
:
-
Clustering
Graph Clustering
BJ Matrix
-
Clustering BJ Matrix
-
Clustering BJ Similarity
BJ
0.0
-
!! ( )
? !
Euclidean Distance Cosine Similarity Jaccard coefficient
manhattan distance, pearson
-
Jaccard : !
Euclidean Distance!
1
2
Jaccard, Euclidean, Cosine
Cosine Similarity!
1 2
-
Euclidean VS Cosine
-
! ?!
-
! !!!
-
! ! ! ! ! ! !! 191 ! 173 !79! ! 46! ! 124!! 129! !56! ! 34! ! 91! ! 59! ! 21! ! 24
-
?
Cosine Similarity - = 0.96966224 - = 0.98190527 ( WIN )
Euclidean Distance - = 85 ( Win ) - = 163
-
?
Cosine Similarity ! ?!Cosine Item !
Euclidean Similarity! ?!Euclidean Item !
-
Eucliean ?
Euclidean
-
Cosine
-
~~~~
-
.
!
TV ?
-
BJ BJ ?
BJBJ
BJ
Euclidean distance : BJ
Cosine Similarity : BJ
-
Clustering BJ Similarity
TV !
Euclidean Distance Cosine Similarity Jaccard coefficient
manhattan distance, pearson
-
Clustering BJ Similarity
BJ
-
Clustering BJ Similarity
BJ1 BJ2 BJ3 BJ4 BJ5 BJ6 BJ7 BJ8 BJ9 BJ10BJ1 0.12 0.1009 0.1582BJ2 0.12 0.0008 0.029 0.0572 0.290BJ3 0.12 0.1391BJ4 0.0854 0.12BJ5 0.12 0.1391 0.0854 0.12 0.0854BJ6 0.290 0.290BJ7 0.12 0.1391 0.0854 0.12BJ8 0.290 0.0854 0.31BJ9 0.0854 0.0854BJ10 0.290
BJ Matrix ( Cosine )
-
Clustering BJ Similarity
?
-
HDFS Redis Cluster
Sorted Set
hdfs files
Data MR
BJ MR
Clustering BJ Matrix
BJ MR
BJ Matrix MySQL
White / Black List /
-
HDFS Redis Cluster
Sorted Set
hdfs files
Data MR
BJ MR
Clustering BJ Matrix
BJ MR
BJ Matrix MySQL
White / Black List /
Hourly Batch Job
-
HDFS Redis Cluster
Sorted Set
hdfs files
Data MR
BJ MR
Clustering BJ Matrix
BJ MR
BJ Matrix MySQL
White / Black List /
?
-
Clustering BJ Matrix
TV BJ
, .
!
?? BJ ?
-
Clustering BJ Matrix
Cowatching Counts
In/Out Stream Counting
70
45
23
5
89
14
2
-
Clustering BJ Matrix
Cowatching Counts
Message Queue
Redis Writer
Redis Writer
Redis Writer
Redis Cluster
Sorted Set
Aggregation Set
Async Processing
In-Memory
-
Clustering BJ Matrix
Message Queue
Redis Writer
Redis Writer
Redis Writer
Redis Cluster
Sorted Set
Async Processing
Cowatching Counts
-
Clustering BJ Matrix
Cowatching Counts
Message Queue
Redis Writer
Redis Writer
Redis Writer
Redis Cluster
Sorted Set
Async Processing
Aggreg
ation
Write
-
Clustering BJ Matrix
Cowatching Counts
Message Queue
Redis Writer
Redis Writer
Redis Writer
Redis Cluster
Sorted Set
Async Processing
Aggreg
ation
Write In
Memory Store
-
Clustering BJ Matrix
Cowatching Counts
Redis SortedSet
instream:{bjid}
outstream:{bjid}
-
BJ BJ Matrix .
-
0.0
0.623573 0.57521 0.34565 0.21736
1.0
BJ = X
-
0.0
0.623573 0.57521 0.34565 0.21736
1.0
1.0
0.0
BJ = X BJ = Y
-
BJ = X BJ = Y ??
0.0
0.623573 0.57521 0.34565 0.21736
1.0
1.0
0.0
-
0.0
0.623573 0.57521 0.34565 0.21736
(1.0,
1.0)
0.0
(0.62, 0.56)
(0.62, 0.41)
(0.62, 0.12
(0.62, 0.03)
(0.57, 0.74)
(0.57, 0.59)
(0.57, 0.48)
(0.57, 0.43)
(0.57, 0.17)
(0.34, 0.72)
(0.34, 0.50)
(0.34, 0.24)
(0.34, 0.09)
(0.21, 0.61)
(0.21, 0.48)
(0.21, 0.01)
-
0.0
0.623573 0.57521 0.34565 0.21736
0.0
(0.62, 0.56)
(0.62, 0.41)
(0.62, 0.12
(0.62, 0.03)
(0.57, 0.74)
(0.57, 0.59)
(0.57, 0.48)
(0.57, 0.43)
(0.57, 0.17)
(0.34, 0.72)
(0.34, 0.50)
(0.34, 0.24)
(0.34, 0.09)
(0.21, 0.61)
(0.21, 0.48)
(0.21, 0.01)
?
(1.0, 1.0)
-
0.0
0.623573 0.57521 0.34565 0.21736
0.0
(0.62, 0.56)
(0.62, 0.41)
(0.62, 0.12
(0.62, 0.03)
(0.57, 0.74)
(0.57, 0.59)
(0.57, 0.48)
(0.57, 0.43)
(0.57, 0.17)
(0.34, 0.72)
(0.34, 0.50)
(0.34, 0.24)
(0.34, 0.09)
(0.21, 0.61)
(0.21, 0.48)
(0.21, 0.01)
(1.0, 1.0)
0.58
0.700.95
0.500.72
0.590.67
-
0.0
0.623573 0.57521 0.34565 0.21736
0.0
(0.62, 0.56)
(0.62, 0.41)
(0.62, 0.12
(0.62, 0.03)
(0.57, 0.74)
(0.57, 0.48)
(0.57, 0.43)
(0.57, 0.17)
(0.34, 0.72)
(0.34, 0.50)
(0.34, 0.24)
(0.34, 0.09)
(0.21, 0.61)
(0.21, 0.48)
(0.21, 0.01)
(1.0, 1.0)
0.58
0.700.95
0.500.72
0.590.67(0.57, 0.59)
-
0.0
0.623573 0.57521 0.34565 0.21736
1.0
1.0
0.0
!
!
!
!
!
!
??
-
.
-
Offline? !
(0.5) .
!
Offline = *0.5
-
0.0
0.623573 0.57521 0.34565 0.21736
1.0
1.0
0.0
!
!
X X
Offline
-
.
-
!
() IRF(Inverse Relation Frequency)
!
BJ Ranking IFR(Inverse Frequency Rank)
-
24 6 67 12 23
0.298 0.201 0.14 0.11 0.0056
Inverse Relation Frequency
0.0335 0.0124 0.0091 0.002 0.00024
-
? ?
-
/ Theme Rank
-
Theme Rank
-
AB
-
AB
-
, BJ,
-
, BJ,
-
, BJ,
-
, BJ,
-
, BJ,
-
AB Test
-
AB Test
: 2.3%
165%
: 6.1%
165%
-
AB Test
: 5.0%
22%
: 6.1% 22%
-
? !
//
-
TV
-
Data Stream
Data Routing
Cache Store
Batch Process
Realtime Process
BJ
Sys
Stat
Hadoop
HQL Jobs
MapReduce
Elastic Search
Workflow Jobs
Message Q
ueue Architecture
Shard Store
None Shard Store
Zookeeper Coordinator
-
Data Stream
Data Routing
Cache Store
Batch Process
Realtime Process
API
Hadoop
HQL Jobs
MapReduce
Elastic Search
Workflow Jobs
Redis Cluster
MySQL
Zookeeper Coordinator
UDP
SenderFile/M
ysql Tailer
Netw
ork Mirroring
ZeroMQ
Thrift
-
Data Stream
Data Routing
Cache Store
Batch Process
Realtime Process
API
Hadoop
HQL Jobs
MapReduce
Elastic Search
Workflow Jobs
Redis Cluster
MySQL
Zookeeper Coordinator
UDP
SenderFile/M
ysql Tailer
Netw
ork Mirroring
ZeroMQ
Thrift
( )
( , )
-
Data Stream
Data Routing
Cache Store
Batch Process
Realtime Process
API
Hadoop
HQL Jobs
MapReduce
Elastic Search
Workflow Jobs
Redis Cluster
MySQL
Zookeeper Coordinator
UDP
SenderFile/M
ysql Tailer
Netw
ork Mirroring
ZeroMQ
Thrift
-Dynamic Data Pipeline // - Workflow scale-out - Async Data
- Broker - ( )
-
Data Stream
Data Routing
Cache Store
Batch Process
Realtime Process
API
Hadoop
HQL Jobs
MapReduce
Elastic Search
Workflow Jobs
Redis Cluster
MySQL
Zookeeper Coordinator
UDP
SenderFile/M
ysql Tailer
Netw
ork Mirroring
ZeroMQ
Thrift
-
Data Stream
Data Routing
Cache Store
Batch Process
Realtime Process
API
Hadoop
HQL Jobs
MapReduce
Elastic Search
Workflow Jobs
Redis Cluster
MySQL
Zookeeper Coordinator
UDP
SenderFile/M
ysql Tailer
Netw
ork Mirroring
ZeroMQ
Thrift
Shard/None Shard Store Topology
/
-
Data Stream
Data Routing
Cache Store
Batch Process
Realtime Process
API
Hadoop
HQL Jobs
MapReduce
Elastic Search
Workflow Jobs
Redis Cluster
MySQL
Zookeeper Coordinator
UDP
SenderFile/M
ysql Tailer
?
-
Data Stream
Data Routing
Cache Store
Batch Process
Realtime Process
BJ
Hadoop
HQL
MapReduce
Theme Ranks (Elastic Search)
Cowatching Counts (Workflow Jobs)
Message Q
ueue Architecture
()
BJ Matrix ()
Zookeeper Coordinator
Sys
Stat
Theme Ranks
-
?
-
!
-
Data Stream
Data Routing
Cache Store
Batch Process
Realtime Process
BJ
Hadoop
HQL Jobs
MapReduce
Elastic Search
Workflow Jobs
Message Q
ueue Architecture
Shard Store
None Shard Store
Zookeeper Coordinator
Sys
Stat
-
Data Stream
Data Routing
Cache Store
Realtime Process
BJ
Elastic Search !
Message Q
ueue Architecture
None Shard Store
Zookeeper Coordinator
Gender/Age
-
?
-
?
// .
.
.
/ .
-
??
-
I know what you want. : http://blog.nemesys.co.kr/tt/entry/LEAP-Motion-%EB%A7%88%EC%9D%B4%EB%84%88%EB%A6%AC%ED%8B%B0-%EB%A6%AC%ED%8F%AC%ED%8A%B8%EC%9D%98-%EA%B7%B8%EB%82%A0%EC
%9D%B4-%EC%9D%B4%EC%A0%9C-%EB%A9%80%EC%A7%80-%EC%95%8A%EC%95%98%EB%8B%A4?category=0
-
. : http://blog.nemesys.co.kr/tt/entry/LEAP-Motion-%EB%A7%88%EC%9D%B4%EB%84%88%EB%A6%AC%ED%8B%B0-%EB%A6%AC%ED%8F%AC%ED%8A%B8%EC%9D%98-
%EA%B7%B8%EB%82%A0%EC%9D%B4-%EC%9D%B4%EC%A0%9C-%EB%A9%80%EC%A7%80-%EC%95%8A%EC%95%98%EB%8B%A4?category=0
: http://mary-ann.tistory.com/610 :
-
.