The Quantified City: Sensing Dynamics in UrbanSetting

Tahar Zanouda, Noora Al Emadi, Sofiane Abbar, and Jaideep Srivastava

Qatar Computing Research Institute

Hamed Bin Khalifa University

Doha, Qatar

{tzanouda, nalemadi, sabbar, jsrivastava}@hbku.edu.qa

ABSTRACTThe world is witnessing a period of extreme growth andurbanization; cities in the 21st century became nervecenters creating economic opportunities and culturalvalues which make cities grow exponentially. With thisrapid urban population growth, city infrastructure isfacing major problems, from the need to scale urbansystems to sustaining the quality of services for citizenat scale. Understanding the dynamics of cities is criticaltowards informed strategic urban planning. This papershowcases QuantifiedCity , a system aimed at under-standing the complex dynamics taking place in cities.Often, these dynamics involve humans, services, and in-frastructures and are observed in different spaces: phys-ical (IoT-based) sensing and human (social-based) sens-ing. The main challenges the system strives to addressare related to data integration and fusion to enable aneffective and semantically relevant data grouping. Thisis achieved by considering the spatio-temporal space asa blocking function for any data generated in the city.Our system consists of three layer for data acquisition,data analysis, and data visualization; each of which em-beds a variety of modules to better achieve its purpose(e.g., data crawling, data cleaning, topic modeling, sen-timent analysis, named entity recognition, event detec-tion, time series analysis, etc.) End users can browsethe dynamics through three main dimensions: location,time, and event. For each dimension, the system ren-

Copyright is held by the author/owner(s).

ders a set of map-centric widgets that summarize theunderlying related dynamics. This paper highlights theneed for such a holistic platform, identifies the strengthsof the ”Quantified City” concept, and showcases a work-ing demo through a real-life scenario.

1. INTRODUCTION’We shape our buildings, and afterwards, our build-

ings shape us.’ wrote Winston Churchill [].After the revolutionary growth of the modern indus-

try in the late 18th century, cities have became a nicheof new opportunities that attracted huge numbers of lo-cal and international migrants. In 2015, we were 53.89%to live in urban areas and this number is projected toreach 70% by the end of 2050. This fast urbaniza-tion has exposed different services and infrastructuresin cities to an increasing stress, leading to a plethora ofurban challenges related to human mobility (e.g., trafficcongestion), public health (e.g, air pollution), neighbor-hood deprivation (e.g., lack of parks), and economicaldifficulties (e.g., unemployment). The situation is suchthat the United Nation has declared managing urbanareas as one of the most important development chal-lenges of the 21st century.

Luckily, the new cities that we are building are bigdata generators, which unleashed many initiatives touse data-driven methods in order to tackle the afore-mentioned challenges and understand more generallyhow cities work and operate [2]. A new body of lit-erature related to Urban Computing has been enabledby the availability of data at large scales [8]. On onehand, the advent of the Internet of Things [1] made iteasier to collect real-time data about the city dynamicsincluding: infrastructure usage, power consumption, en-vironment, health related issues, human behavior, andcommutes. At the other hand, the high penetrationof smart connected devices (e.g., smartphones) and therapid adoption of social media platforms, has turned hu-

arX

iv:1

701.

0425

3v1

[cs

.SI]

16

Jan

2017

mans into mobile and real-time data generators, lead-ing to what is known as social sensing [7]. Thus, re-searchers have studied different types of dynamics bymining spatio-temporal data generated either by usersor sensors. Examples include using call data records(CDR) to understand mobility and commute patternsin cities [3, 4], using crowd-sourced taxi GPS data toestimate gas consumption and air pollution [6], and fi-nally using check-ins data from location-based socialnetworks (LSBNs) to infer traffic congestion [7].

Using IoT-based sensing data has long time tradi-tion in building smart city applications such as parkingmanagement systems, whereas using social-based sens-ing data has been extensively used in building life styleapplications such as recommender engines for restau-rants. We aim in this paper to bridge the gap betweenthese two spaces, by designing QuantifiedCity – acomprehensive platform that makes it possible to lever-age data from different spaces to build smarter appli-cations. QuantifiedCity aims at quantifying the dy-namics involving humans, services and infrastructures,taking place in cities. A good example to showcase therelevance of this combination are road traffic monitoringsystems. Indeed traditional systems rely on the deploy-ment of physical sensors on road segments and inter-sections to count the number of cars passing by. Thisallows a nice ”real-time” visualization of the traffic sta-tus in different parts of the city. However, relying solelyon physical sensors, makes it impossible to proceed withthe diagnosis of the traffic. A red congested segment,can be so for many reasons (e.g., accident, malfunction-ing traffic light, weather condition, etc.) that are com-pletely hidden to the system. The social space providesa nice complementary as many users may have postedonline in a completely unsolicited way about the eventcausing the traffic. Thus, combining the physical spacewith the social space, modulo the deployment of full-fledged intelligence to link data in the two spaces, mayresults in semantically richer monitoring dashboards.QuantifiedCity consists of three layers for data

acquisition, data analysis, and data visualization. Itsmain contribution is the use of the spatio-temporal spaceas a blocking function to fuse data that is generated indifferent spaces (e.g., physical, social). This is inspiredfrom the work of Jaro [5] in which he introduced theuse of blocking methods for record linkage. The intu-ition behind our adaptation is to say that two pieces ofinformation are related if they are generated in samespatial location and within the same time interval. Thesecond big contribution of QuantifiedCity is the useof cross-modal data mining and machine learning tech-niques in order to better combine features from the dif-ferent spaces of data in the knowledge discovery pro-cesses [8].

We describe in the following sections the global archi-tecture of QuantifiedCity and provide some detailsabout each of its layers. We also include a demonstra-tion of its capabilities in a real-life scenario.

SocialData

InternetofThings

Humanmobility

EventDetection

TrafficInference

Locationidentificat

ion

ContentAnalysis

SQL Mongo

Streamacquisition

Insights

Spark

Statistics

Recomm-nedations

Rheem HTML5CSS3NodeJS

DataAcquisition DataAnalysis DataVisualization

Functio

nalities

Technologies

Mapping

Giraph

Figure 1: Overview of data extraction and datafusion Approach

2. THE QUANTIFIED CITYThe general architecture of the QuantifiedCity is

composed of three major components: (1) data acqui-sition, (2) data analysis, and (3) data visualization.The data acquisition component contains the neededmodules to scrap and aggregate streams of data, aswell as static geographical data sets such as census.Data streams are processed concurrently in real-timeand stored in one database.

The data analysis component contains modules thatanalyze the data. The results of the analysis are visual-ized through in a map-centric dashboard. The systemis designed as a suite of independently, small, scalablemodules to make it easier and simpler for development.

In what follows, we provide the details of each com-ponent.

2.1 Data AcquisitionThis module allows the collection and fusion of sev-

eral data sets. As a first step, we acquire static geo-graphical and census data 1, this later is used to spa-tially index the city’s data. In order to fuse the knowl-edge from different data sources, we use spatio-temporalunits to conduct our studies. We divide the city intohigh resolution spatial units, and the day on fine-grainedtime units to make it easier to study an urban phenom-ena from different angels. As an example: when westudy the effects of on-road traffic on urban air pollu-tion, it’s important to collect data on air quality from

1http://www.mdps.gov.qa/ar/Pages/default.aspx

CurrentEvent

QatarExxonMobilOpen

Figure 2: Main GUI of QuantifiedCity . This showcases some statistics around ”Qatar ExxonMobilOpen” tennis tournament such as overall sentiment per location (map) and time (time line), densityof interactions on the map, traffic status, etc.

environmental sensors, number of cars from proximitysensors, and events happening in the same neighbor-hood from social streams. The cross-referenced datasetscan provide insights on relationship between the choiceof local events’ places and long-term air quality level inthe city. This later can lead urban planners to conclu-sions that can not be identified easily.

Our system captures, concurrently and continuously,data streams on the city’s infrastructure and on thecity’s social interactions. Once captured, data streamsare indexed based on time, and geographical informa-tion. Then, data is loaded to centralized geo-database.

2.2 Data analysisThis module consists of a set of data mining and

machine learning algorithms to enable knowledge dis-covery from cross-modal fused data. This set of intel-ligence techniques vary from content-based techniques(sentiment, NER, topic modeling, event detection, etc.),to structure-based techniques such as network analysis(community detection, etc. ).

As an example, the system is able to capture and minesocial content, detect events, and sense the sentiment ofpeople. By mining social networks, and photo-sharingplatforms, the system can reveal places that attract peo-ple most. Using content-analysis, we can identify placesfrom text, link activity to places and verify whetherpeople complain about traffic using sentiment analysis.Our results can optimize traffic light system in the city

to effectively manage big events.

2.3 Data visualizationAs we have different mediums of information to show-

case, a highly interactive system is needed to allow ef-ficient and smooth interactions with complex intercon-nected data. The visualization explorer enables usersto explore the data easily and intuitively. It is mainlycomposed of two sub-views, namely, (1) the dashboard,(2) map view.Dashboard View: The dashboard view shows the re-

sults of the stats information. The view shows differentpages, categorized in main categories and can be ac-cessed by scrolling from top to bottom. Every pagehas different panels, and every panel contain a chart tovisually communicate results of the different analysis.Map view: Using Map-centred exploratory approach

allows users to easily select spatial choices, as it intu-itively provides contextual insights. In addition to themap layer, we use colored zones shapes to showcase thedifferent results. The zones layer enables users to spec-ify the targeted zone.

3. DEMONSTRATIONIn this section, we showcase a detailed demonstra-

tion plan and describe how different users (monitors,urban planners, decision makers, police, etc.) can in-teract with QuantifiedCity .

The system provides contextual insights based on threedimensions: i. time, ii. location, and iii. event. As datain our system is indexed to a particular time span, userscan select the beginning and the end of the time range,and the granularity of the time analysis (e.g., groupdata by 20 secs or 5 minutes.) Users are able to nar-row focus to a subset of data based on the selected timebounds. Similarly, users can select specific regions or aparticular location to filter the data and have limitedviews to their selections. More interestingly, users cantap into the wealth of data around one event. Subse-quently, users can combine different geographical andtemporal criteria to gain access to intuitive, yet insight-ful reports.

As an example to illustrate the results, we will fo-cus on some big Tennis tournaments that took placein Qatar this January and show how QuantifiedCitycaptures the urban dynamics (social activity, sentimentanalysis, fans mobility, traffic congestion, etc.) aroundthese events. With the use of content-analysis meth-ods, QuantifiedCity can identify and distinguish be-tween events in the same context (i.e., “Tennis tour-nament” will be automatically associated with QATAREXXONMOBIL OPEN for men during the first weekof January, and will be linked to Qatar Total Tourna-ment for women in the last week, thanks to the spatio-temporal blocking function). The system is able tofuse the knowledge from different contextual data toestimate crowd movement patterns (from sensor dataabout cars moving around the stadiums, and social me-dia activity and engagement), and evaluate the com-plaints to the city managers during the event (usingtopic modeling and sentiment analysis). The informa-tion can be used to fuel a recommendation engine tohelp re-routing people from-to the game venues, it canalso notify the authorities about the need of an evacu-ation plan if needed. The system can be connected tothe road traffic light system to adjust crowd movement.

Figure 2 shows QuantifiedCity system visualizingresults of Doha city. On the right side of the dash-board, various panels allow users to see the differentresults of data analysis modules. On the left side, amap with zones polygons are visualized. By clicking onevery zone, the system will generate statistics that willtake the same form, but projected on one zone.

4. CONCLUSIONIn this paper, we presented QuantifiedCity , a sys-

tem that captures the complex dynamics taking placein fast growing cities. The system relies on a compre-hensive set of data sources coming from different lay-ers: physical overlay through IoT-based sensing and hu-man overlay through social-based sensing which enablesa holistic understanding of how cities work and oper-ate. QuantifiedCity brings a novel perspective intodata integration paradigm by using the spatio-temporalspace as a blocking function to link parcels of data com-

ing from different sources. The intuition being that datagenerated in the same geographical space and time win-dow can be grouped together and used for knowledgediscovery related to what is happening in that location.By bridging the gap between the physical and the socialsides of cities, and through an interactive and intuitivemap-centric interface, our system allows urban plannersand decision makers to visually inspect problems, andinteractively analyze and track in real-time the devel-opment of events.

5. REFERENCES[1] L. Atzori, A. Iera, and G. Morabito. The internet

of things: A survey. Computer networks,54(15):2787–2805, 2010.

[2] M. Batty, K. W. Axhausen, F. Giannotti,A. Pozdnoukhov, A. Bazzani, M. Wachowicz,G. Ouzounis, and Y. Portugali. Smart cities of thefuture. The European Physical Journal SpecialTopics, 214(1):481–518, 2012.

[3] M. C. Gonzalez, C. A. Hidalgo, and A.-L. Barabasi.Understanding individual human mobilitypatterns. Nature, 453(7196):779–782, 2008.

[4] S. Grauwin, S. Sobolevsky, S. Moritz, I. Godor, andC. Ratti. Towards a comparative science of cities:Using mobile traffic records in new york, london,and hong kong. In Computational approaches forurban environments, pages 363–387. Springer, 2015.

[5] M. A. Jaro. Advances in record-linkagemethodology as applied to matching the 1985census of tampa, florida. Journal of the AmericanStatistical Association, 84(406):414–420, 1989.

[6] J. Shang, Y. Zheng, W. Tong, E. Chang, andY. Yu. Inferring gas consumption and pollutionemission of vehicles throughout a city. InProceedings of the 20th ACM SIGKDDInternational Conference on Knowledge Discoveryand Data Mining, KDD ’14, pages 1027–1036, NewYork, NY, USA, 2014. ACM.

[7] S. Wang, L. He, L. Stenneth, P. S. Yu, and Z. Li.Citywide traffic congestion estimation with socialmedia. In Proceedings of the 23rd SIGSPATIALInternational Conference on Advances inGeographic Information Systems, SIGSPATIAL’15, pages 34:1–34:10, New York, NY, USA, 2015.ACM.

[8] Y. Zheng, L. Capra, O. Wolfson, and H. Yang.Urban computing: concepts, methodologies, andapplications. ACM Transactions on IntelligentSystems and Technology (TIST), 5(3):38, 2014.