Current Progress of Big Data Utilization for Official Statistics In Indonesia

Ali SaidDirector of Analysis and Development of StatisticsBPS-Statistics Indonesia

Presented at Asia-Pacific Stats Café Series, 17 August 2020

Legal Aspect of BPS-Statistics Indonesia in Utilizing Big Data as Data Source

Section II, Article 7 states:

“Statistics are organized through data collection by:

➢ census;

➢ survey;

➢ administrative product compilation; and

➢ other ways according to the development of science and technology”

Big Data is a product of development of science and technology

Act No. 16 Year 1997 on Statistics

Legal Aspect for Big Data Access (E-Commerce Data)

Government Regulation No. 80/2019 regarding “E-Commerce Organizers”

Ministry of Trade Regulation No. 50/2020 reagarding “Provisions on Business Licensing, Advertising, Guidance and Supervision of Businesses Trading Trade through Electronic Systems”

Article 21 poin (f): “Domestic and/or foreign e-commerce organizers must provide data and/or information regularly to government agency organizing government affairs in the field of statistics”

Article 14: “Domestic and/or foreign e-commerce organizers must provide dataand/or information regularly to government agency organizing government affairs in the field of statistics according to the provisions of the legislation”

BPS-Statistics Indonesia is preparing the regulation regarding the mechanism of data and information provision by e-commerce organizers

BPS Data Architecture

Enterprise Data Warehouse

Output Data

Warehouse Data Mart

Dissemination Data

Warehouse Data Mart

Administrative

Census/Survey

External Web

Spatial

Data Sources Collection System BI Tool

Landing

Zone

Working

Zone

GoldZone

ELT

Real Time- DB Sream ProcessMachine-Deep

Learning

Consistent Security & Governance

Data Storage

Processing

Cloud

Pack for

Data

Insight

ETL

Internal System

Integrated

Dissemination

SystemMacro-

Analysis

System

Frame

Register

SystemELT

Enterprise Data Lake as single source of truth

External

System

Data Operating System

Others

BPS Roadmap of Big Data Development

2014-2016

2017

• Exploration different Big Data Sources

• Visibility study of Big Data• Searching the possibility of

Big Dta use for Official Statistics (MPD)

• Proof of Concepts:- price monitoring- mobility tracking

• Development of Text Mining (Bahasa Indonesia)

Proof of Concepts:➢ Price monitoring (different

sources)➢ Mobility behavior tracking➢ Text Mining for different

Official Statistics

2018

2019

• Initial Development of Big Data Infrastructure

• Image Analysis for different purposes

• E-commerce Analysis• Big Data experts development

• Development of Big Data Infrastructure phase 2

• producing statistical Indicators using e-commerce data and other big data sources

• War room implementation• QAF development for Big

Data

• Development of Big Data Infrastructure phase 1

• Satellite Images Analysis• Automatic E-commerce

Analysis• Exploring other big data

sources to produce statistics• Larger Big Data experts• War room development

• Extensifying big data uses by exploring new Big Data sources

• Implementing artificial intelligence in statistical activities

• Revolution in Data Gathering, analysis and Dissemination

• New sources of Big Data?

2020

2021

2022 2024

2023

• Scaling up the

utilization of Big Data

for many purposes by

implementing QAF

• Initiation of artificial

intelligence

development

• Intensifying the utilization

of Big Data to produce

quality statistics or

information to support

official statistics

• Artificial intelligence

development

Big Data Utilization for Statistics

Big Data utilization in BPS-Statistics

Indonesia

Complementary statistics to support official statistics

New statistics to respond real time data need

As official statistics

For improving sampling frame

KAB. BANDUNG

BARAT KAB. BANDUNG

KAB. SUMEDANG

Use of MPD in BPS-Statistics IndonesiaInternational Tourism Statistics /Inbound Tourism (implemented for official stat)

Domestic Tourism Statistics (on progress)

Commuter Statistics

Other Statistics (Forthcoming research)

Metropolitan Statistical Area (MSA) Development (in progress)

• 2016: 19 districts• Since 2017: 25 districts• Filtering and calibration is improved• Improved Methodology

• Demand for domestic tourism data at district level• Piloting in 2018 and 2019: hh survey for domestic tourism VS MPD& Digital survey• Digital survey based on selected sample indicated by MPD [expenditure, motivation, etc]• Output: O-D matrix of 514 districts

• MPD will replace conventional survey on commuters• Commuters and circular travellers are the by product of domestic tourism MPD

Commuter Survey MPD

• Employment statistics• Social statistics• Transporation statistics

• Delineation of metropolitan area of Bandung Raya• Other metropolitan areas [East Java and Sumatra]

BANDUNG CITY

CIMAHI CITY

Big Data for New Statistics to Respond COVID-19 and Provide Supporting Information for Official Statistics

CHANGES IN POPULATION MOBILITY Google Mobility Index(1)

Population mobility and activity at home rose dramatically after the implementation of WFH policy on 17 March and appealing people to stay at home to respond the COVID-19 pandemic, but after WFO was implemented in 5 June (transition period) mobility at home is declining and remains stable until

now due to the policy allowing 50% of employees to work from home

CHANGES IN POPULATION MOBILITY BY PROVINCE(Google Mobility Index) (2)

At Home

Population mobility at home during the pandemic increased dramatically especially in some provinces in Java and Bali, while mobility at work, in contrast, dropped in .

At Work

Source: Processed from https://www.google.com/covid19/mobility

Note: Percentage change against baseline (Feb 2020) Note: Percentage change against baseline (Feb 2020)

Muslim fsetiv dayMuslim fsetiv day

CHANGES IN POPULATION MOBILITY Google Mobility Index(3)

10

Mobility at Retail and Recreation

WFH Muslim Festiv day WFO

After WFH implementation policy, mobility at retail and recreation rose significantly, but after Muslim festival day, mobility at retail and recreation tends to increase but the figure still below

the pandemic

“

Mobility at Grocery and Pharmacy

WFH WFO

Few days towards Eid/Muslim festival day, people mobility at grocery increased dramatically and did not care about the pandemic“

Mobility at Transits

WFH WFO

Since the COVID pandemic, poeple tended to reduce their outgoing activity using public transport , even after the implementation policy on WFO“

Mobilitya at Park

WFH WFO

People tend to avoid going to the park during the pandemic“

Muslim Festiv day

11

FLIGHT CONDITION INDONESIA, 15 MARCH – 30 JUNE 2020

Based on data from flight tracker,air transport activity droppedsubstantially during the pandemicin 5 busiest airports

Complimentary information for official

statistics

• Total product sold by classification

• Average price by type of products

• Number of shops (account) by regions

• Ect.

Other information generated from market place:

MARKET PLACE ANALYSIS

13

Analysis of Big Data on EmploymentJob Vacancy Advertisement[1]

No. of job vacancy advertisements, Sept 2019-May 2020 Number of companies offering job vacancies, September 2019-May 2020

✓ During Sept 2019-March 2020, number of job vacancies wererecorded at relatively high ranging from 10,000 to 12,500. Butduring the pandemic, the figure declined to just below 4000.

✓ More than 500 companies offering job vacancies through jobs.idsite every month, but after the pandemic the figure continued todecline

Online site of jobs search mainly used as reference by job seekers

First Covid case officially annouced

Big Data to Produce Complimentary Information for Unemployment Figure

Analysis of Big Data on EmploymentJob Vacancies Advertisement[2]

Total job

vacancies

advertisement

by sector

During the period January–May 2020, the number of job vacancyadvertisements at all sectors tended to decline

14

1962

1846

1792

1610

1475

1120

474

465

454

970

Jan Feb March AprilInformation & Communication

Jasa Perusahaan

Financial and Assurance Services

Manufacturing Industry

Other Services

Trade

Health and Social Activity

Services

Education Services

Accommodation and Restourant

Other Sectors

1711

1340

1605

1480

1391

976

449

496

386

975

1577

1301

1473

1552

1562

1098

498

447

548

1047

1186

556

1134

636

1014

492

366

143

138

469

868

305

614

398

454

512

176

105

33

261

May

1.308

A

14.169

B

33.218

A=B

A = Data from Web ScrapingB = BPS Statistics Indonesia Data Frame C = Matched data

Big Data for Improving Data Frame

Case study:Food and Beverage service activities data using

google map for Java and Bali

A

C

B

Other On-going Big Data Research

Use of marketplace data for calculating Consumer Price Index

Use of google trend for predicting quarterly data of fixed capital formation

Use of google trend for predicting unemployment rate

Concluding Remark

Big Data is promising to support or produce official statistics, but we

must work hard to overcome big challenges

THANK YOU

“Big data is at the foundation of all the megatrends that are happening.”

Chris Lynch, American Writer of Books