combining big data with small data for the best … · 2019-03-29 · combining mobile positioning...
TRANSCRIPT
![Page 1: COMBINING BIG DATA WITH SMALL DATA FOR THE BEST … · 2019-03-29 · Combining mobile positioning data (big data) and survey (small data) provide the best result of official statistics](https://reader034.vdocuments.net/reader034/viewer/2022050213/5f5fa9b37ba52d5e1c03d86f/html5/thumbnails/1.jpg)
COMBINING BIG DATA WITH SMALL DATA FOR THE BEST RESULTS OF OFFICIAL STATISTICS
![Page 2: COMBINING BIG DATA WITH SMALL DATA FOR THE BEST … · 2019-03-29 · Combining mobile positioning data (big data) and survey (small data) provide the best result of official statistics](https://reader034.vdocuments.net/reader034/viewer/2022050213/5f5fa9b37ba52d5e1c03d86f/html5/thumbnails/2.jpg)
2
1 Inbound Tourism
![Page 3: COMBINING BIG DATA WITH SMALL DATA FOR THE BEST … · 2019-03-29 · Combining mobile positioning data (big data) and survey (small data) provide the best result of official statistics](https://reader034.vdocuments.net/reader034/viewer/2022050213/5f5fa9b37ba52d5e1c03d86f/html5/thumbnails/3.jpg)
STATISTICS INDONESIA
3
LIMITATION OF MOBILE POSITIONING DATA
Lack of Qualitative Data (such as purpose of visits,
accommodation type).
Expenditure.
Mobile phone usage (mobile phone usage
behaviour).
Small data (survey data) is required to obtain data that can
not be provided by mobile positioning data
![Page 4: COMBINING BIG DATA WITH SMALL DATA FOR THE BEST … · 2019-03-29 · Combining mobile positioning data (big data) and survey (small data) provide the best result of official statistics](https://reader034.vdocuments.net/reader034/viewer/2022050213/5f5fa9b37ba52d5e1c03d86f/html5/thumbnails/4.jpg)
4
Riau islands (SG border) Borneo (MY)
Sintang&Talaud (PH)
Papua (PNG)
East Nusa Tenggara (TL)
![Page 5: COMBINING BIG DATA WITH SMALL DATA FOR THE BEST … · 2019-03-29 · Combining mobile positioning data (big data) and survey (small data) provide the best result of official statistics](https://reader034.vdocuments.net/reader034/viewer/2022050213/5f5fa9b37ba52d5e1c03d86f/html5/thumbnails/5.jpg)
Malaysia
Indonesia
Immigration entry points
100 km
Ka
limant
an
-Born
eo
Roamer trip start areahigh low
![Page 6: COMBINING BIG DATA WITH SMALL DATA FOR THE BEST … · 2019-03-29 · Combining mobile positioning data (big data) and survey (small data) provide the best result of official statistics](https://reader034.vdocuments.net/reader034/viewer/2022050213/5f5fa9b37ba52d5e1c03d86f/html5/thumbnails/6.jpg)
STATISTICS INDONESIA
Survey was conducted in 25 Kabupaten cover 76 gates, July & October 2017, samples= 36.000
Cross Border Survey to obtain:
Mobile phone Usage During Crossing Border
Purpose of Travel
Expenditure
Main Occupation
6
CROSS-BORDER SURVEY
![Page 7: COMBINING BIG DATA WITH SMALL DATA FOR THE BEST … · 2019-03-29 · Combining mobile positioning data (big data) and survey (small data) provide the best result of official statistics](https://reader034.vdocuments.net/reader034/viewer/2022050213/5f5fa9b37ba52d5e1c03d86f/html5/thumbnails/7.jpg)
STATISTICS INDONESIA
7
BORDER SURVEY QUESTIONNAIRE
How many phones
How many SIM cardsCountry of residence
Purpose of visit
Length of stay
Frequency of visit
Occupation
Nationality
Expenditure
![Page 8: COMBINING BIG DATA WITH SMALL DATA FOR THE BEST … · 2019-03-29 · Combining mobile positioning data (big data) and survey (small data) provide the best result of official statistics](https://reader034.vdocuments.net/reader034/viewer/2022050213/5f5fa9b37ba52d5e1c03d86f/html5/thumbnails/8.jpg)
WHAT IS MPD? DATA OVER- AND UNDERCOVERAGE
Indonesian resident
Not roaming Roaming
Foreign SIM
Foreign resident
Transit Worker Tourist
Roaming
Foreign SIM
Multiple SIM
Single SIM
Not roaming
Local SIM card
Switch off roaming
Switch off phone
No cell phone
MPD
No
t in
MP
D -
ad
d
Not tourist - exclude
Exclude >1
Overcoverage
Undercoverage
![Page 9: COMBINING BIG DATA WITH SMALL DATA FOR THE BEST … · 2019-03-29 · Combining mobile positioning data (big data) and survey (small data) provide the best result of official statistics](https://reader034.vdocuments.net/reader034/viewer/2022050213/5f5fa9b37ba52d5e1c03d86f/html5/thumbnails/9.jpg)
STATISTICS INDONESIA
FILTERING MPD AT BORDER GATES
AT : Additional Tourists based on MPD
MPD : Number of sim card foreigner detected by MNO
Xroam : Ratio sim card per mobile phone
Pnr : Ratio of foreigner Non-roaming (using local sim card or no mobile phone)
MS : Market share
WCI : Inbound Tourism recorded by Immigration
9
𝑨𝑻 =𝑴𝑷𝑫
𝑿𝒓𝒐𝒂𝒎×
𝟏
𝟏 − 𝑷𝒏𝒓×
𝟏
𝑴𝑺− 𝑾𝑪𝑰
![Page 10: COMBINING BIG DATA WITH SMALL DATA FOR THE BEST … · 2019-03-29 · Combining mobile positioning data (big data) and survey (small data) provide the best result of official statistics](https://reader034.vdocuments.net/reader034/viewer/2022050213/5f5fa9b37ba52d5e1c03d86f/html5/thumbnails/10.jpg)
WHAT IS MPD? DATA OVER- AND UNDERCOVERAGE
Indonesian resident
Not roaming Roaming
Foreign SIM
Foreign resident
Transit Worker Tourist
Roaming
Foreign SIM
Multiple SIM
Single SIM
Not roaming
Local SIM card
Switch off roaming
Switch off phone
No cell phone
MPD
PN
R R
ati
o
P Ratio
Xroam Ratio
Overcoverage
Undercoverage
LOS1 Filter Pw Ratio
![Page 11: COMBINING BIG DATA WITH SMALL DATA FOR THE BEST … · 2019-03-29 · Combining mobile positioning data (big data) and survey (small data) provide the best result of official statistics](https://reader034.vdocuments.net/reader034/viewer/2022050213/5f5fa9b37ba52d5e1c03d86f/html5/thumbnails/11.jpg)
Bengkalis remain Bengkalis dropped
1 Stay in Bengkalis waters >6h
2 Enter Bengkalis inland
3 Go to Sumatra
1 Go straight to Batam
2 Go straight to Karimun
3 Go to Dumai
0
10000
20000
30000
40000
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
2018 2019
VISITORS GRAPH
0
10000
20000
30000
40000
50000
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
2018 2019
DROPPED SEAMEN GRAPH
![Page 12: COMBINING BIG DATA WITH SMALL DATA FOR THE BEST … · 2019-03-29 · Combining mobile positioning data (big data) and survey (small data) provide the best result of official statistics](https://reader034.vdocuments.net/reader034/viewer/2022050213/5f5fa9b37ba52d5e1c03d86f/html5/thumbnails/12.jpg)
TTU remain TTU dropped
1 Go to Atambua3 Use Atambua airport
1 Accidental Roaming
in one cell
2 Seamen
2 Go to nearby ID towns
0
10000
20000
30000
40000
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
2018 2019
VISITORS GRAPH
0
5000
10000
15000
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
2018 2019
DROPPED SEAMEN AND ACCIDENTAL ROAMING GRAPH
3 TL transit
![Page 13: COMBINING BIG DATA WITH SMALL DATA FOR THE BEST … · 2019-03-29 · Combining mobile positioning data (big data) and survey (small data) provide the best result of official statistics](https://reader034.vdocuments.net/reader034/viewer/2022050213/5f5fa9b37ba52d5e1c03d86f/html5/thumbnails/13.jpg)
TTU remain TTU dropped
![Page 14: COMBINING BIG DATA WITH SMALL DATA FOR THE BEST … · 2019-03-29 · Combining mobile positioning data (big data) and survey (small data) provide the best result of official statistics](https://reader034.vdocuments.net/reader034/viewer/2022050213/5f5fa9b37ba52d5e1c03d86f/html5/thumbnails/14.jpg)
MPD is useful, it gives
more accurate and
coverage compare to
cross border survey.
However, there also
limitation of MPD.
Survey is also useful to
obtain ground truth and
qualitative and
quantitative data that
can not be obtain from
MPD
Lesson Learnt
14
So, there is no
data source that
is superior to
other data
source. All of
data source could
complete each
other
![Page 15: COMBINING BIG DATA WITH SMALL DATA FOR THE BEST … · 2019-03-29 · Combining mobile positioning data (big data) and survey (small data) provide the best result of official statistics](https://reader034.vdocuments.net/reader034/viewer/2022050213/5f5fa9b37ba52d5e1c03d86f/html5/thumbnails/15.jpg)
CONCLUSION
Combining mobile positioning data (big data) and survey (small data)
provide the best result of official statistics
survey
1
2
15
MPD
Formula for additional tourism at the border
3
𝑨𝑻 =𝑴𝑷𝑫
𝑿𝒓𝒐𝒂𝒎×
𝟏
𝟏 − 𝑷𝒏𝒓×
𝟏
𝑴𝑺− 𝑾𝑪𝑰
Combining mobile positioning data (big data) and administrative data
(immigration data) also provide the best result of official statistics
![Page 16: COMBINING BIG DATA WITH SMALL DATA FOR THE BEST … · 2019-03-29 · Combining mobile positioning data (big data) and survey (small data) provide the best result of official statistics](https://reader034.vdocuments.net/reader034/viewer/2022050213/5f5fa9b37ba52d5e1c03d86f/html5/thumbnails/16.jpg)
16
2 Domestic Tourism and
Commuting
![Page 17: COMBINING BIG DATA WITH SMALL DATA FOR THE BEST … · 2019-03-29 · Combining mobile positioning data (big data) and survey (small data) provide the best result of official statistics](https://reader034.vdocuments.net/reader034/viewer/2022050213/5f5fa9b37ba52d5e1c03d86f/html5/thumbnails/17.jpg)
STATISTICS INDONESIA
17
COMPARISON (1)
Household Survey (Conventional)
Distribution sample 30.000 census blocks,
50.000 travellers
Listing, chose eligible sample, estimate population
face to face interview
Using MPD & digital survey
Distribution sample All census block (more
than 800.000)
+/- 140 million subscribers
All domestic tourism subscribe to MNO, estimate for the non subscribers
Ground check
![Page 18: COMBINING BIG DATA WITH SMALL DATA FOR THE BEST … · 2019-03-29 · Combining mobile positioning data (big data) and survey (small data) provide the best result of official statistics](https://reader034.vdocuments.net/reader034/viewer/2022050213/5f5fa9b37ba52d5e1c03d86f/html5/thumbnails/18.jpg)
STATISTICS INDONESIA
18
COMPARISON (2)
Household Survey
Minimum : 8.000 x 4 = 32.000 man
survey Recruitment Training Questionnaire & manual Office supplies
Quarterly, memory lapse
Sampling problem
Resistant from respondent
Remote area problem, difficult to be reached
No enumerators are needed, reduce work burden
Monthly (timeliness), even daily data
Fact, real data(accuracy)
MPD & Digital Survey
![Page 19: COMBINING BIG DATA WITH SMALL DATA FOR THE BEST … · 2019-03-29 · Combining mobile positioning data (big data) and survey (small data) provide the best result of official statistics](https://reader034.vdocuments.net/reader034/viewer/2022050213/5f5fa9b37ba52d5e1c03d86f/html5/thumbnails/19.jpg)
One subscriber Tourism Trip
Home
WorkTourism Trip
![Page 20: COMBINING BIG DATA WITH SMALL DATA FOR THE BEST … · 2019-03-29 · Combining mobile positioning data (big data) and survey (small data) provide the best result of official statistics](https://reader034.vdocuments.net/reader034/viewer/2022050213/5f5fa9b37ba52d5e1c03d86f/html5/thumbnails/20.jpg)
STATISTICS INDONESIA
20
CONNECTION BETWEEN THE DIGITAL SURVEY & MPD
ONE SUBSCRIBER TOURISM TRIP POPULATION
FOR EACH SUBSCRIBER:
1. REMOVE USUAL ENVIRONMENT; 2. APPLY TOURISM ALGORITHM;3. MAKE A LIST OF NUMBER THAT
DO TOURIST TRIPS;4. THIS LIST IS A SAMPLING FRAME
FOR THE DIGITAL SURVEY
SAMPLE
FOR SUBSCRIBER WHO TRAVEL
1. SEND SMS; 2. ASK THEIR PROFILE;3. ASK WHETHER THEY VISITED
CERTAIN AREA;4. ASK FOR PURPOSE OF TRIPS,
TRANSPORTATION, ACCOMMODATION, EXPENDITURE
![Page 21: COMBINING BIG DATA WITH SMALL DATA FOR THE BEST … · 2019-03-29 · Combining mobile positioning data (big data) and survey (small data) provide the best result of official statistics](https://reader034.vdocuments.net/reader034/viewer/2022050213/5f5fa9b37ba52d5e1c03d86f/html5/thumbnails/21.jpg)
DOMESTIC VISITOR – DIGITAL SURVEY TO COMPLETE BIG DATA
21
![Page 22: COMBINING BIG DATA WITH SMALL DATA FOR THE BEST … · 2019-03-29 · Combining mobile positioning data (big data) and survey (small data) provide the best result of official statistics](https://reader034.vdocuments.net/reader034/viewer/2022050213/5f5fa9b37ba52d5e1c03d86f/html5/thumbnails/22.jpg)
DOMESTIC VISITOR – DIGITAL SURVEY TO COMPLETE BIG DATA
22
![Page 23: COMBINING BIG DATA WITH SMALL DATA FOR THE BEST … · 2019-03-29 · Combining mobile positioning data (big data) and survey (small data) provide the best result of official statistics](https://reader034.vdocuments.net/reader034/viewer/2022050213/5f5fa9b37ba52d5e1c03d86f/html5/thumbnails/23.jpg)
STATISTICS INDONESIA
23
THE SANDBOXING
SCENARIO A (AT THE VERY BEGINNING)
BPS TEAM WENT TO MNO AND DID SANDBOXING THERE MEMBER OF THE TEAM: DATA SCIENTIST, METHODOLOGY STAFF, TOURISM S TATISTIC STAFF
SCENARIO B (LATER ON)
BPS TEAM CONNECT TO MNO SERVER THROUGH VPNWE NEED TO REQUEST FOR AN ACCESS AND RENEW THE ACCESS EVERY 4 (F OUR) HOURS.
SCENARIO C (WHAT WE WISH FOR)
MNO HAND THE DATA OVER TO BPS TEAM
![Page 24: COMBINING BIG DATA WITH SMALL DATA FOR THE BEST … · 2019-03-29 · Combining mobile positioning data (big data) and survey (small data) provide the best result of official statistics](https://reader034.vdocuments.net/reader034/viewer/2022050213/5f5fa9b37ba52d5e1c03d86f/html5/thumbnails/24.jpg)
THANK YOU