InterSCity: A Scalable Microservice-based Open Source Platform forSmart Cities

Arthur de M. Del Esposte1, Fabio Kon1, Fabio M. Costa2 and Nelson Lago1

1Department of Computer Science, University of Sao Paulo, R. do Matao, 1010 - Cidade Universitaria, 05508-090, SaoPaulo, Sao Paulo, Brazil

2Institute of Informatics, Federal University of Goias, Alameda Palmeiras, Quadra D, Campus Samambaia, 74690-900,Goiania, Goias, Brazil

{esposte, kon, lago}@ime.usp.br, fmc@inf.ufg.br

Keywords: Smart Cities, Software Platform, Microservices, Scalability, Open Source Software

Abstract: Smart City technologies emerge as a potential solution to tackle common problems in large urban centersby using city resources efficiently and providing quality services for citizens. Despite the various advancesin middleware technologies to support future smart cities, there are no universally accepted platforms yet.Most of the existing solutions do not provide the required flexibility to be shared across cities. Moreover,the extensive use and development of non-open-source software leads to interoperability issues and limitsthe collaboration among R&D groups. In this paper, we explore the use of a microservices architecture toaddress key practical challenges in smart city platforms. We present InterSCity, a microservice-based opensource smart city platform that aims at supporting collaborative, novel smart city research, development, anddeployment initiatives. We discuss how the microservice approach enables a flexible, extensible, and looselycoupled architecture and present experimental results demonstrating the scalability of the proposed platform.

1 INTRODUCTION

The rapid growth of cities around the world has cre-ated large, densely populated urban centers charac-terized by complex interconnected structural, socialand economic organizations. This urbanization phe-nomenon imposes several challenges for sustainabledevelopment and quality of life in cities that tra-ditional management approaches cannot overcome.Thus, smart cities emerge as a new paradigm aimedat addressing these problems by using city resourcesefficiently and providing quality services for its cit-izens. As a consequence, several efforts have beendevoted to the study and development of smart citysolutions to address the major problems faced by thecities of today such as traffic jams, air pollution, andenergy efficiency.

Smart cities are characterized by the adoption ofInformation and Communication Technologies (ICT)as an integral part of the city’s infrastructure to sup-port multiple solutions for urban challenges (Neirotti

∗This research is part of the INCT of the Future Internetfor Smart Cities funded by CNPq, proc. 465446/2014-0,CAPES, proc. 88887.136422/2017-00, and FAPESP, proc.2014/50937-1.

et al., 2014). The Internet of Things (IoT), Big Data,and Cloud Computing are key enabling technologiesof smart cities that offer a wide range of opportuni-ties and challenges, both in the academy and industry.To fully exploit the potential of these enablers, futuresmart cities will demand a unified ICT infrastructureto properly share their resources rather than relyingon non-integrated solutions, or market islands (Vil-lanueva et al., 2013). The most common approachstill adopted in existing initiatives is to develop ad-hoc applications for specific domains, such as healthcare and urban mobility, which leads to low resourcesharing and the proliferation of non-interoperable ser-vices.

Many authors advocate that integrated middle-ware platforms can provide the required infrastructureto support the construction of sophisticated, cross-domain smart city applications (Villanueva et al.,2013; Hernandez-Munoz et al., 2011; Fazio et al.,2012). Such platforms must include facilities for ap-plication development, for enabling interoperabilitybetween the different systems of a city, for managinglarge amounts of data, and for dealing with a hetero-geneous number of distributed devices and services atcity scale, to cite only a few requirements (Santana

et al., 2016).Despite the various advances in middleware tech-

nologies, protocols, and standards, many aspects re-lated to the design, development, deployment, andmanagement of smart city platforms still challengethe research community. Consequently, there are nouniversally accepted platforms yet, and existing so-lutions do not provide the required flexibility to beshared across cities (PCAST, 2016). In the following,we highlight three key factors that contribute to thelack of practical and reusable solutions in the field.

First, in addition to traditional functional capabili-ties, smart city platforms must meet a number of non-functional requirements to enable their use in differ-ent environments and applications. Security and pri-vacy policies may change according to the laws andregulations of the city, impacting design decisions re-garding storage and availability of the data in the plat-form. Different contexts may expose a great diversityof requirements, which may dynamically evolve overtime. Thus, smart city platforms must provide a flex-ible architecture to adopt new technologies and sup-port new functional and non-functional requirementsto suit the diversity of the multiple and constantlyevolving city environments where they are deployed.Similarly, a smart city platform must scale up to han-dle the increasingly large number of users, devices,and services as well as their associated data, whichwill grow with the evolution of urban environments.In particular, these platforms must offer different scal-ability strategies by design to meet the diverse scala-bility demands. Therefore, the dynamism and con-tinuous evolution of urban environments require theuse of new approaches to develop flexible, evolvable,maintainable, and scalable architectures for smart cityplatforms (Krylovskiy et al., 2015).

The second point is related to the lack of prac-tical and scientific validation to evaluate the differ-ent aspects of smart city solutions, such as social andeconomic impact, internal and external quality, per-formance, scalability, and feasibility, to cite a few.In (Sanchez et al., 2011), the authors observed thatalthough many IoT projects present concrete solu-tions, validation of the developed technologies andarchitectural models are limited to proofs-of-concept,not allowing conclusive results. Therefore, we stillneed more comprehensive studies with experimentsand tests that allow the comparison of different smartcity technologies. For this purpose, significant effortshould be devoted to (1) deploying existing solutionsin real production scenarios, (2) developing meth-ods and tools for more sophisticated simulation-basedevaluations, and (3) developing well-defined bench-mark strategies for cross-platform evaluation.

Lastly, the extensive use and development of non-open-source software in the core of smart city plat-forms jeopardize their widespread adoption as it leadsto interoperability difficulties and limits the collabora-tion among R&D groups, often forcing them to “rein-vent the wheel”. The use of open technologies is cru-cial to the sustainability and development of futuresmart cities, since they prevent vendor lock-in, enablecollaborative development, market opportunities, andsharing of solutions.

To address the above-mentioned challenges, wepropose InterSCity, an open source microservices-based platform for smart cities. Its objective is to pro-vide a high-quality, modular, highly scalable middle-ware infrastructure to support smart city solutions thatcan be reused across cities and R&D groups, as wellas governments and companies.

Microservice architectures emerged from the soft-ware industry’s best practices in building large-scaledistributed applications composed of small, intercon-nected components (microservices), supporting scal-ability, evolvability, maintainability, and modularity.InterSCity leverages the microservice approach to im-plement the fundamental modules described by thereference architecture proposed in (Santana et al.,2016), conceived from the analysis of 23 smart cityprojects. This reference architecture describes thebuilding blocks needed to meet the main functionaland non-functional requirements to guide the devel-opment of next-generation platforms for smart cities.Thus, the platform provides high-level services tomanage heterogeneous IoT resources, data storageand processing, and context-aware resource discov-ery. In this paper, we discuss the design and imple-mentation details of the InterSCity platform to ad-dress the design challenges in smart city systems.

This paper brings two key contributions to smartcities platforms research: (I) the InterSCity platformas an open-source project to enable novel smart cityresearch, development, and deployment initiatives;(II) an analysis of the impact of its microservice ar-chitecture to address key research challenges relatedto scalability and evolvability in smart city platformsbased on experimental results and our early experi-ence. Section 2 presents a discussion of related work.Section 3 describes in detail the InterSCity microser-vice architecture, design principles, and an exampleapplication. Section 4 presents a scalability evalua-tion of the platform within a scenario of data stream-ing from sensors spread across a city. Lastly, wepresent the concluding remarks and future work inSection 5.

2 RELATED WORK

Several efforts have been devoted to the study anddevelopment of platforms that address the key chal-lenges of smart cities. In particular, projects that aimat addressing the practical problems related to the de-velopment, deployment, and maintenance of smartcity services and applications are the most relevant inthe context of this work.

Perhaps the most notable project that targets therealistic deployment and validation of smart city so-lutions is SmartSantander, which provides a smartcity testbed with research facilities composed of morethan 20,000 IoT devices deployed in urban envi-ronments (Sanchez et al., 2014). The SmartSan-tander testbed aims at supporting experimentationwith smart city services in a realistic setting at a largescale. Despite the relevance of this project to enableexperiments in real scenarios, it is not clear whetherit meets important requirements that would allow itsapplicability in other contexts (e.g., in cities with dif-ferent characteristics), such as adaptability, flexibility,and extensibility. We advocate that an open sourcesmart city platform based on a scalable microservicesarchitecture is an effective way to achieve such re-quirements.

A number of middleware platforms were devel-oped in recent years to address multiple requirementstowards the construction of cross-domain smart citysolutions, as opposed to traditional approaches basedon vertical silos. The Civitas middleware (Villanuevaet al., 2013) fulfills the main functional requirementsby proposing a set of essential standards and tools toenable smart city ecosystems. The Gambas (Apoli-narski et al., 2014) project offers tools to facilitate thedevelopment and deployment of smart city applica-tions, including a runtime environment and an SDK.However, it is not clear whether these platforms ad-dress some of the key non-functional requirements,such as scalability, extensibility and evolvability.

OpenIoT1 is one of the most relevant projects han-dling the main requirements of smart city platforms.It is an open source layered middleware platform thataims at enabling semantic interoperability across IoTapplications, including smart cities (Soldatos et al.,2015). The platform provides visual tools to facili-tate the administration and implementation of appli-cations directly on top of it. Although OpenIoT pro-vides several facilities to support IoT applications, itis not clear how its architecture deals with importantaspects related to scalability, adaptability, and exten-sibility.

Krylovskiy et al. (Krylovskiy et al., 2015) present

1https://github.com/OpenIotOrg/openiot

the DIMMER project, a microservice-based IoT plat-form to support applications that aim at improvingenergy efficiency and management in cities. This isthe only previous work we have found that exploresthe use of a microservices architecture to build smartcity platforms. The authors described their early ex-perience in adopting microservices, covering their im-pact on organizational and design aspects. However,their work leaves several open questions that demandfurther experimental and empirical studies to obtainmore conclusive results on the adoption of microser-vices in the design of smart city platforms, such ason issues related to scalability and perfomance. Ourwork has a broader scope, as the InterSCity platformis designed to support smart city applications frommultiple domains and is fully developed as an opensource project, as opposed to the DIMMER platform.

3 THE INTERSCITY PLATFORM

InterSCity is a project aiming at developing multi-disciplinary, high-quality scientific and technologicalresearch to address the key challenges related to thesoftware infrastructure of smart cities (Batista et al.,2016). One of the main goals of InterSCity is to de-velop reusable open source technologies and methodsto support future Smart Cities. In this context, the In-terSCity platform is a concrete result from the initialresearch efforts in the project.

The InterSCity platform has been developed in-crementally as an open source microservices-basedplatform to enable collaborative research, develop-ment, and experiments in smart cities. The platformwas designed from the outset following the refer-ence architecture for smart city platforms proposedin (Santana et al., 2016). This reference architectureaims at guiding the development of next-generationsmart city platforms by describing and organizing themajor building-blocks that are required to meet a widerange of functional and non-functional requirementselicited from the analysis of a large number of existingsmart city projects. By implementing key building-blocks of this architecture, the InterSCity Platformcovers the major features required to support inte-grated smart city applications in different domains,such as public transportation, public safety, and envi-ronmental monitoring. Currently, the InterSCity Plat-form provides a set of high-level cloud-based servicesto manage heterogeneous IoT resources, data storageand processing, and context-aware resource discov-ery.

We follow agile software development methodsaligned with practices of open-source project com-

munities to provide a high-quality platform that canbe shared and collaboratively developed among cities,research groups and development communities. Thesource code is available online 2.

3.1 Design Principles

Smart cities emerge from advances in tools and tech-niques developed both in the industry and academia,such as the Internet of Things, Big Data, and CloudComputing. However, the integration of these toolsand technologies is not straightforward, as these ar-eas evolve rapidly and their combination raises com-plex issues. This brings new challenges, approaches,and dynamics that result in the constant emergenceof novel technologies, standards, and services. Smartcity applications further enhance the dynamism of theinvolved technologies, since they encompass com-plex environments composed of several intercon-nected subsystems which are constantly evolving andpresenting new challenges. Such dynamism impactsseveral design decisions in smart city platform archi-tectures.

To provide a high-quality, practical smart cityplatform to support future smart city projects, wemust address two key design issues that jeopardize thewide adoption of existing solutions in different smartcity initiatives: scalability and evolvability

Scalability - A platform must scale well in mul-tiple dimensions to properly support a smart city.Among others, smart city platforms must handle: (I)a large number of devices that compose the city IoTinfrastructure; (II) millions of users and componentsthat use the platform services; (III) a very large vol-ume of city-related data that must be stored and pro-cessed; (IV) a potentiality large set of new servicesthat can interact with the platform to offer comple-mentary capabilities. The scalability requirementsmay vary depending on the context and should be ad-dressed since the beginning of any smart city project.

Evolvability - Urban environments are very dy-namic and tend to change constantly in terms of or-ganization, regulations, problems, opportunities, andchallenges. Therefore, smart city platforms must beadaptable to meet changing requirements in a cost-effective way. Evolvability is the system’s capabilityto evolve over time by supporting rapid modificationand enhancement with low cost and small architec-tural impact, and is a fundamental element for thesuccess and economic value of long-lived software(Breivold et al., 2012)

Our strategy to address scalability and evolvabilityissues and to provide the required services to support

2https://gitlab.com/smart-city-software-platform

smart cities is to adopt a microservices architecturefor the platform. The microservices model emergedfrom the software industry efforts to build large-scaledistributed systems refining SOA guidelines throughthe adoption of DevOps and Agile principles, tools,and techniques. Although there are efforts to under-stand the impact of this architecture on other researchareas (Le et al., 2015; Gopu et al., 2016), very fewworks explore the potential of microservices in thecontext of smart cities.

Here we present the design principles adoptedin the InterSCity Platform which are aligned withmicroservices patterns, which are critical for thewide adoption of the platform in different smart cityprojects.

• Modularity via Services. Modularity is a keyconcept used in software architecture to dividesystems into smaller functional units. Microser-vice architectures achieve modularity throughsingle-purpose, small services that communicatethrough lightweight mechanisms to achieve acommon goal.

• Distributed Models and Data. In monolithicarchitectures and even in traditional service-oriented systems, it is fairly common to create aunified domain model and a centralized storagebackend. With microservices, each service has itsown database and models, which may evolve in-dependently of external services. Decentralizeddata management and the possibility to use dif-ferent technologies that best fit each context arerelevant advantages. On the other hand, increasedoperational complexity is the main drawback.

• Decentralized Evolution. Microservices mustbe autonomous, providing well-defined bound-aries and communication APIs so that they canevolve and be maintained independently. More-over, this principle ensures that each service mayimplement its functionalities using the most ap-propriate technology, provoking positive technol-ogy heterogeneity. Similarly, each microservicemay scale independently using different strate-gies, since scalability requirements vary acrossservices. Finally, this design principle should re-flect on the configuration and deployment proce-dures, which may be performed independently aswell.

• Reuse of Open Source Projects. Reusing soft-ware components is a fundamental practice ofsoftware engineering to achieve productivity, costeffectiveness and software reliability. We alwaysgive preference to the use of existing robust open-source tools, libraries, and frameworks instead of

implementing components from scratch, since thequality of popular open-source packages is admit-tedly good as already empirically observed (Taibi,2013). Moreover, we adopt a rigorous technol-ogy selection criteria and only incorporate opensource components that have an active developercommunity, stable release support, and appropri-ate documentation to guide usage, development,and deployment.

• Adoption of Open Standards. As important asthe reuse of open-source projects is the adop-tion of open, well-accepted standards that are de-signed to provide interoperability at different lev-els. This prevents technology and vendor lock-in.The use of open Internet and web standards is es-sential to enable the true Internet of Things, beingwidely used in related projects found in the lit-erature (Fazio et al., 2012; Amaral et al., 2015;Hernandez-Munoz et al., 2011).

• Asynchronous versus Synchronous. Althoughmost services provide RESTful APIs, they mustimplement asynchronous messaging as muchas possible to avoid blocking in synchronousrequest-reply interactions. Asynchronicity shouldbe achieved by using notifications, the pub-lish/subscribe design pattern, and event-basedcommunication strategies to support low latencyand scalability. Besides, the platform must rely ona lightweight message bus with the single purposeof providing a reliable messaging service ratherthan traditional SOA approaches that use sophis-ticated, heavy middleware such as an EnterpriseService Bus (ESB).

• Stateless Services. This design principle supportsscalability by advocating that services should bestateless to enable any service instance to re-spond to any request, facilitating load distributionand elasticity. Thus, the design of microservicesshould separate state data, such as context and ses-sion data, to be managed by an external compo-nent whenever possible.

3.2 Platform Architecture

The InterSCity microservices architecture is shownin Figure 1, addressing important aspects of IoT andData management, providing high-level RESTful ser-vices to support the development of smart city appli-cations, services and tools for different purposes. Theunderlying IoT Gateways can register new devices tothe platform and send sensor data through a RESTAPI. InterSCity abstracts the complexity involved inthe communication between smart city applications

and IoT devices, as well as the complexity of city-scale data management.

Figure 1: The InterSCity Platform Architecture.

InterSCity provides well-defined boundaries tocommunicate with both IoT devices and smart city ap-plications. Currently, the platform is composed of sixdifferent microservices that provide features for theintegration of IoT devices (Resource Adaptor), dataand resource management (Resource Catalog, DataCollector and Actuator Controller), resource discoverythrough context data (Resource Discovery) and visu-alization (Resource Viewer). Although all microser-vices expose REST APIs for synchronous messagingover HTTP, most of the communication for the com-position of services is done through asynchronouscalls, relying on the lightweight message bus Rab-bitMQ3 for asynchronous messaging through the Ad-vanced Message Queuing Protocol (AMQP).

Interactions between the platform and its clientsinvolve the manipulation of city resources. A city re-source is a logical concept that encapsulates a physi-cal entity that makes up the city, such as cars, buses,traffic lights, and lampposts. Resources comprise at-tributes (e.g., location and description) and functionalcapabilities to provide data and receive commands,which are respectively supported by sensors and ac-tuators coupled to the resource. This approach facili-tates the interaction of client applications with a realcity environment, since it grants an abstraction com-posed of city concepts rather than the cyber-physicalparticulars, that comprise the underlying IoT layersof Smart City ecosystems. As a consequence, for in-stance, two buses registered in the platform are ac-cessed through the same standardized API regard-

3www.rabbitmq.com

less of their devices and communication technologies.Likewise, two physical sensor or actuator deviceswith similar purposes are encapsulated as a commoncapability, abstracting all the specific details related todata representation and deployment of these devices.

The InterSCity architecture supports a decentral-ized management of city resources dividing func-tional responsibilities and data persistence across themicroservices. The Resource Adaptor microserviceworks as a stateless proxy that allows external ser-vices, such as IoT gateways, to register and update re-sources on the platform, post sensed data from thoseresources, and subscribe to events that indicate actua-tor commands. All city resources are registered in theResource Catalog microservice which is responsiblefor providing universally unique identifiers (UUIDs)(Leach et al., 2005) and for asynchronously notifyingthe event of resource creation to other microservices.

The Data Collector microservice stores sensor datacollected by city resources. Sensor data consist ofcontext information or an event linked to a resourcecapability which is observed at a particular time. DataCollector provides an API to allow access to both cur-rent and historical context data of city resources usinga rich set o filters that can be accessed in search end-points. The Actuator Controller microservice, in turn,provides standardized services to intermediate all ac-tuation requests to city resources with actuator capa-bilities. Moreover, Actuator Controller records the his-tory of actuation requests so that they can be accessedin the future, e.g., for auditing purposes.

Both the Resource Discovery and the ResourceViewer microservices provide more sophisticated ser-vices by orchestrating Data Collector and ResourceCatalog. Resource Viewer is a web visualization mi-croservice for presenting city resources informationgraphically based on Resource Catalog and Data Col-lector back-end services. The purpose of ResourceViewer is to present general and administrative visu-alizations of city resources, including location, realtime context data, and representative charts of histor-ical data. The Resource Discovery microservice pro-vides a context-aware search API that may be usedby client applications to discover available city re-sources. This API provides filters that can be com-bined to discover resources. For instance, filters cancombine information such as location data, intervalrules for current context data, and other types of meta-data.

Scalability is a key non-functional requirementfor smart city platforms. InterSCity was primarilydesigned to support smart city projects with a largeamount of users, data, and services. Its microservicesarchitecture supports scalability at the functional de-

composition level in both application and databasetiers by splitting the processing responsibilities acrossseveral services, dividing the amount of data into de-centralized databases, and isolating services throughfine-grained interfaces. However, each microservicewill be required to scale at different paces depend-ing on both the specificities of the urban and tech-nological context and the continuously increasing de-mands. Although the loosely coupled architecture al-lows microservices to scale out independently, dif-ferent design strategies must be applied to overcometheir own traits in order to achieve horizontal scala-bility. Table 1 summarizes the scalability strategiescurrently supported by InterSCity microservices, de-noted by and points out new strategies that will besupported in the near future, denoted by NF.

A deployment of the InterSCity platform mayhave several instances of each of its microservicesbehind Load Balancers to handle higher loads trans-parently for customers. Services that receive asyn-chronous messaging, such as Data Collector, Actua-tor Controller, and Resource Adaptor, are designed tosupport the addition of more background workers tohandle highly intensive demands for event-based jobs.Data Collector also uses database caching supportedby Redis4 to provide low-latency readings of the lat-est data collected by city resources. Resource Dis-covery caches static resource meta-data provided bythe Resource Catalog, since they do not change veryoften. Initially, we adopted PostgreSQL5 in all mi-croservices as it is widely adopted in the software in-dustry and it supports georeferenced queries, whichare important in the smart city domain. However, weintend to move towards plurality of database systemsin a near future to better fit the scope of each microser-vice and to properly support horizontal scalability sothat the database layer does not become a bottleneck.

The loosely coupled message-oriented communi-cation approach favors the continuous development ofthe platform, enabling the extension of existing fea-tures through service composition and the additionof new microservices to meet the constantly evolvingsmart city requirements. Moreover, decoupled com-munication interfaces allow us to maintain microser-vices in separate code repositories enabling: decen-tralized version and dependency management, inde-pendent, faster tests, safe refactoring, evolution of ex-isting features, and the adoption of the most appropri-ate technologies in each context. The lower boundaryprovided by Resource Adaptor enables InterSCity tocontinuously integrate heterogeneous IoT technolo-gies without affecting other services. It can also be

4www.redis.io5www.postgresql.org

Table 1: Scalability strategies supported by InterSCity microservices

Microservice HTTP Load Balancer Background Workers Caching Database ShardingResource AdaptorResource Catalog NF

Data Collector NFActuator ControllerResource Discovery

Resource Viewer NF

used to integrate the existing legacy ICT infrastruc-ture of cities, such as open data initiatives. It is worthnoting that InterSCity’ upper API isolates client ap-plications from the modification or addition of newtechnologies in the smart city infrastructure.

InterSCity microservices architecture requiresDevOps methods to support both an agile develop-ment life cycle and consistent, automated, indepen-dent deployments. Thus, we encapsulated microser-vices into individual Docker 6 lightweight containerswhich can be deployed and maintained independently.Continuous integration tools play an important rolein the automated execution of both individual and in-tegration tests, and to ensure that container imagesare built correctly along the development of microser-vices. To perform automated, consistent, remote de-ployments, we adopted the Ansible7 automation en-gine, which provides a set of configuration manage-ment tools and scripts, facilitating the deployment ofthe InterSCity platform and associated applications.

3.3 Application Life-cycle

To demonstrate the aforementioned approaches, herewe illustrate how applications can be built over the In-terSCity services to interact with city resources. Forthis purpose, we exemplify the use case of the SmartParking App, an experimental smart city applicationdeveloped on top of the platform by University of SaoPaulo students during a graduate course and whosesource code is available in the platform distribution.The Smart Parking application aims to help the diffi-cult task of finding available parking spots in a largecity, by offering a map with geolocated real-time in-formation of parking spaces. The system is based onboth static and sensor data of individual parking spotsequipped with embedded sensors to notify the pres-ence of a parked car. As can be seen in Figure 2, theSmart Parking App allows drivers to discover closeparking spaces by offering visualization filters relatedto their availability, prices, and operating hours. Onecan check the details of a parking spot and view the

6www.docker.com7www.ansible.com

route from the user’s current location, as shown inFigure 3. The hardware part of this example was sim-ulated via a specific software component that mim-icked the behavior of physical sensors.

Figure 4 illustrates a message flow that resultedfrom the use of the Smart Parking application sup-ported by the InterSCity platform hosted in a cloud in-frastructure. This example considers a smart parkinginfrastructure supported by cyber-physical systems todetect the presence of cars in parking spaces based ontechnologies that are commonly used in smart parkingsolutions, such as Wireless Sensor Networks (WSN),Light Dependable Resistor (LDR) sensors, Infra-Red(IR) sensors, and magnetic sensors (Bachani et al.,2016). These sensors send data continuously to aremote IoT Gateway via wireless protocols, such asZigBee or Bluetooth, as illustrated in Step 1. Theresponsibilities of the IoT Gateway are two-fold: (I)registering each connected parking spot as a city re-source with the “availability” sensor capability (Step2) and (II) notifying the platform when a parkingspace becomes available or unavailable (Step 3). Forthese purposes, the IoT Gateway must track resourceUUIDs provided by InterSCity to be able to send con-text data through the Resource Adaptor API uponstate change events. We highlight that the conceptsof city resource and capability abstract the implemen-tation details of the underlying WSN infrastructure.

To use The Smart Parking application, one mustdefine a target location or automatically use his/hercurrent GPS data in addition to setting custom param-eters to filter parking spaces according to the desiredcharacteristics (Step 4). As an example, the applica-tion may query the platform for all the parking spaceresources that match the selected parameters within a500 meters radius of the target location through theResource Discovery API (Step 5). As a result, theSmart Parking application renders the current state ofthe returned parking resources on the map, as shownin Figure 2. Users can set the update time intervalto get the current state of the returned resources aswell as to modify the parameters, which will resultin new requests such as the one performed in Step 5.Additional requests may be performed to get detailed

Figure 2: Screenshot of the Smart Parking application.

Figure 3: Parking spot details in the Smart Parking applica-tion.

information about a specific parking spot from the Re-source Catalog API, such as presented in Figure 3, oreven to obtain its availability history through the DataCollector microservice (Step 6).

4 PERFORMANCE ANDSCALABILITY ANALYSIS

To evaluate the proposed platform, we conducted twopreliminary experiments. First, we evaluated how the

performance of the InterSCity platform degrades withthe increase in the number of concurrent IoT gateways(clients) sending sensor data continuously over a longperiod of time. The main objective of this experimentwas to evaluate the individual behavior and consump-tion of hardware resources of each microservice sothat we could identify potential bottlenecks. The sec-ond experiment aimed at assessing the scalability ofthe platform by applying supported scalability strate-gies to the bottlenecks we identified.

To conduct the experiments, we ran a production-like instance of the InterSCity platform in the DigitalOcean 8 cloud. Both InterSCity microservices andexternal services, such as PostgreSQL, RabbitMQ,and Redis, were deployed within Docker containers.However, each service instance was hosted on its ownvirtual machine for isolation purposes, guaranteeing afixed amount of machine resources per service. In theexperiments, we consider a common smart city sce-nario where distributed sensors continuously collectobservations from the city and send the sensed datato IoT gateways. The source code of the scripts usedin the experiment are available in the repository forreproducibility 9.

8www.digitalocean.com9https://github.com/LSS-USP/smart-city-platform-

Figure 4: Smart Parking application life cycle.

4.1 Degradation Analysis

For the first experiment, we used a total of four single-core GNU/Linux Debian 8.6 machines with 512MBRAM having 2.0GHz of clock speed in the same pri-vate network hosting one single instance of the Re-source Adaptor, Resource Catalog, Data Collector,and RabbitMQ services, without any replication. Weperformed the degradation analysis by benchmark-ing the platform against 11 different workloads sup-ported by the Funkload10 load testing tool. Workloadswere characterized by the number of concurrent em-ulated IoT Gateways that continuously send sensordata from city resources to the platform. Each client(the IoT Gateways) runs a loop sending synchronousrequests to the platform throughout the experiment asfast as it can. We ran each of the load tests for fourminutes, with a 30-second interval between them, col-lecting both response time and throughput metrics,while keeping the same capacity and configuration ofthe platform during all workload tests. We repeatedeach run of the experiment 20 times.

Figure 5 shows the performance degradation ofthe platform as the number of concurrent IoT Gate-ways increases. The best average response time oc-

experiments10funkload.nuxeo.org

curs for a workload of 50 concurrent clients, whichwas less than 60 milliseconds.

The average response time remained below 1 sec-ond with a workload of up to 350 parallel clients.However, the tests with 250 or more clients in parallelstart to have requests with latency above 1 second; forthis particular application, this is not a problem, butthis is an interesting indication of the limitations ofthe platform in case of applications with more strin-gent real-time requirements. It is important to notethat the number of failed requests (returned with anerror code or with a timeout) varied from 0.01% for350 concurrent clients to 0.16% for 600 concurrentclients. With up to 250 concurrent IoT Gateways,100% of the requests were successful.

Figure 5: Response time degradation.

In addition to evaluating the degradation in theoverall performance, we also monitored the use ofhardware resources on each machine to identify po-tential bottlenecks. For this purpose, we used theLinux-based Collectl 11 tool, as it can be used to mon-itor a broad set of subsystems such as CPU, disk,memory, processes, and network. Among the fourmachines used to deploy the platform in the experi-ment, the one that hosted the Resource Adaptor mi-croservice presented the highest load in CPU andmemory usage, both close to 100% during the entireduration of the experiment, being identified as the firstbottleneck in the tested scenario. The Data Collectorhost machine also maintained a high level of CPU us-age, as well as an intensive use of I/O operations tostore the sensed data. The other two machines did notcharacterize bottlenecks.

11collectl.sourceforge.net

4.2 Scalability Analysis

The objective of the second experiment was to evalu-ate the scalability of the InterSCity platform, as wellas to demonstrate the flexibility of our architectureto address scalability issues. The same smart cityscenario was considered, with concurrent IoT gate-ways continuously sending sensor data from city re-sources to the platform. For this experiment we kepta fixed workload with 500 concurrent clients to evalu-ate the platform’s speedup and scale-up metrics. Thespeedup metric measures how the performance im-proves with the addition of new resources to the sys-tem, while the scale-up metric measures the through-put gain.

The loosely coupled microservices architecture al-lows us to increase only the resources of the identi-fied bottleneck microservices. We benchmarked theplatform with the fixed workload during six 4-minutecycles applying a round-robin load balancing strat-egy by adding a replica of the Resource Adaptor mi-croservice for each new cycle. The first cycle usedexactly the same deployment setup of the first experi-ment described before. Both the Load Balancer (NG-INX12) and the new Resource Adaptor instances weredeployed on Docker containers hosted by separatesingle core, GNU/Linux Debian 8.6, 512MB RAM,2.6GHz machines. These tests were performed usingthe Apache Benchmark13 Linux tool.

As a result of the scaling strategy, the average re-sponse time decreased from 1725 milliseconds (with1 instance) to 320 milliseconds (with 6 instances).Figure 6 shows the performance improvement mea-sured in the experiment. We can observe a signifi-cant performance gain when scaling horizontally onlyone of the microservices that make up the platform;the speedup is very close to optimal. Since all mes-sages received by Resource Adaptors are publishedthrough the RabbitMQ message service, the use ofCPU by this service increases considerably, indicatinganother possibility for improving the speedup even alittle more. RabbitMQ offers horizontal scalabilitynatively and we plan to enhance the platform makinguse of this feature, further improving its scalability.

As can be seen in Figure 7, the mean through-put of the platform increased substantially by hori-zontally scaling the Resource Adaptor in the testedscenario. With 6 instances, the platform answered anaverage of 1546 requests per second, 5.5 times morethan when using the configuration with a single Re-source Adaptor. Similarly to the speedup metric, thescale-up value increases almost at the same rate at

12https://www.nginx.com/13httpd.apache.org/docs/2.4/programs/ab.html

Figure 6: Speedup - performance improvement varying thenumber of Resource Adaptors.

which new instances are added, which is an excellentresult.

Figure 7: Throughput improvement varying the number ofResource Adaptors.

5 CONCLUSION AND FUTUREWORK

Smart city platforms play a key role in the devel-opment of future smart cities as they support cross-domain solutions, interoperability among multiplecity systems, and resource and data sharing. How-ever, due to various technical, practical, and method-ological challenges, the community still lacks robustsolutions that can be shared across smart city initia-tives, as well as production environments to supportscientific validation of existing proposals. This pa-per presents two key contributions: (I) advances inthe state-of-art by exploring the impact of a microser-vices architecture in the design, development and de-ployment of scalable smart city platforms; and (II) anovel microservice-based open source smart city plat-form that provides facilities to build next-generation

scalable smart city solutions and to integrate hetero-geneous IoT systems.

Our early experience with the development of In-terSCity shows that microservices can be properlyused to build smart city solutions that provide finer-grained, single purpose building blocks that can bemore easily and independently evolved compared totraditional SOA approaches. However, it also intro-duces new challenges due to an increase in the overallcomplexity. We highlight that the use of industry stan-dards, such as DevOps techniques and open sourcetools, automated tests, and design patterns, is essentialfor the successful implementation of our project. Inthe near future, we intend to investigate more deeplythe impact of applying microservices design princi-ples to achieve a loosely coupled, evolvable architec-ture, demonstrating that InterSCity can be adapted fordifferent smart city environments, can easily integratenew services, and can be modified to meet the con-stantly evolving city requirements.

Experimental results point towards the applicabil-ity of our approach in the context of smart cities, sincethe platform can support different scalability demandswhile keeping acceptable performance. These resultsalso show that microservices can be deployed andscaled independently. Further comprehensive exper-iments should be performed to evaluate all the ser-vices provided by the platform. More specifically,we intend to conduct experiments to evaluate the per-formance and scalability of the InterSCity platformwithin more realistic scenarios of smart cities, withdevices, data, and users at a larger city scale.

Our ongoing work includes several features stillneeded to meet the constantly evolving requirementsof urban environments. This includes more sophisti-cated Big Data processing and analytics (Al Nuaimiet al., 2015) as well as improved data visualization.

The adopted open source model encourages thecommunity to take advantage of our contribution, aswell as to contribute to the InterSCity platform evolu-tion. We expect to enable future smart city initiativesand research from other groups on new approaches tobuild open, high-quality, practical solutions that canbe extended, reused, collaboratively evolved, and de-ployed in real smart city environments.

ACKNOWLEDGEMENTS

We acknowledge the student members of the SmartParking App group who develop the application ontop of the InterSCity platform: Debora Setton, HansHarley, Jefferson Silva, Nury Arosquipa, and ThiagoPetrone.

We also acknowledge the following developers fortheir contributions to the InterSCity platform sourcecode: Alander Marques, Ariel Palmeira, Arthur DelEsposte, Athos Ribeiro, Cadu Elmadjian, Caio Sal-gado, Caroline Satye, Danilo Caetano, Debora Setton,Fernando Freire, Henrique Potter, Igor Lima, JoaoBrito, Leonardo Pereira, Lucas Kanashiro, MacarturSousa, Marisol Solis, Rodolfo Scotolo, Rodrigo Faria,Rodrigo Siqueira, Rogerio Cardoso, Thiago Petrone,and Wilson Kazuo.

REFERENCES

Al Nuaimi, E., Al Neyadi, H., Mohamed, N., and Al-Jaroodi, J. (2015). Applications of big data to smartcities. Journal of Internet Services and Applications,6(1).

Amaral, L. A., Tiburski, R. T., de Matos, E., and Hessel, F.(2015). Cooperative middleware platform as a servicefor internet of things applications. In Proceedings ofthe 30th Annual ACM Symposium on Applied Comput-ing, SAC ’15, pages 488–493, New York, NY, USA.ACM.

Apolinarski, W., Iqbal, U., and Parreira, J. X. (2014). Thegambas middleware and sdk for smart city applica-tions. In 2014 IEEE International Conference onPervasive Computing and Communication Workshops(PERCOM WORKSHOPS), pages 117–122.

Bachani, M., Qureshi, U. M., and Shaikh, F. K. (2016). Per-formance analysis of proximity and light sensors forsmart parking. Procedia Computer Science, 83:385– 392. The 7th International Conference on AmbientSystems, Networks and Technologies (ANT 2016) /The 6th International Conference on Sustainable En-ergy Information Technology (SEIT-2016) / AffiliatedWorkshops.

Batista, D. M., Goldman, A., Hirata Jr., R., Kon, F., Costa,F. M., and Endler, M. (2016). Interscity: Addressingfuture internet research challenges for smart cities. In7th International Conference on the Network of theFuture. IEEE.

Breivold, H. P., Crnkovic, I., and Larsson, M. (2012).A systematic review of software architecture evolu-tion research. Information and Software Technology,54(1):16 – 40.

Fazio, M., Paone, M., Puliafito, A., and Villari, M. (2012).Heterogeneous sensors become homogeneous thingsin smart cities. In Innovative Mobile and Internet Ser-vices in Ubiquitous Computing (IMIS), 2012 Sixth In-ternational Conference on, pages 775–780.

Gopu, A., Hayashi, S., Young, M. D., Kotulla, R., Henschel,R., and Harbeck, D. (2016). Trident: scalable com-pute archives: workflows, visualization, and analysis.volume 9913, pages 99131H–99131H–12.

Hernandez-Munoz, J. M., Vercher, J. B., Munoz, L.,Galache, J. A., Presser, M., Gomez, L. A. H., and Pet-tersson, J. (2011). The future internet. chapter Smart

Cities at the Forefront of the Future Internet, pages447–462. Springer-Verlag, Berlin, Heidelberg.

Krylovskiy, A., Jahn, M., and Patti, E. (2015). Designinga smart city internet of things platform with microser-vice architecture. In Future Internet of Things andCloud (FiCloud), 2015 3rd International Conferenceon, pages 25–30.

Le, V. D., Neff, M. M., Stewart, R. V., Kelley, R.,Fritzinger, E., Dascalu, S. M., and Harris, F. C. (2015).Microservice-based architecture for the nrdc. In 2015IEEE 13th International Conference on Industrial In-formatics (INDIN), pages 1659–1664.

Leach, P. J., Mealling, M., and Salz, R. (2005). A uni-versally unique identifier (uuid) urn namespace. RFC4122, RFC Editor. http://www.rfc-editor.org/rfc/rfc4122.txt.

Neirotti, P., Marco, A. D., Cagliano, A. C., Mangano, G.,and Scorrano, F. (2014). Current trends in smart cityinitiatives: Some stylised facts. Cities, 38:25–36.

PCAST (2016). Technology and the future of cities, reportto the president. Technical report, Executive Office ofthe President, United States.

Sanchez, L., Galache, J. A., Gutierrez, V., Hernandez, J. M.,Bernat, J., Gluhak, A., and Garcia, T. (2011). Smart-santander: The meeting point between future internetresearch and experimentation and the smart cities. In2011 Future Network Mobile Summit, pages 1–8.

Sanchez, L., Munoz, L., Galache, J. A., Sotres, P., San-tana, J. R., Gutierrez, V., Ramdhany, R., Gluhak, A.,Krco, S., Theodoridis, E., and Pfisterer, D. (2014).Smartsantander: Iot experimentation over a smart citytestbed. Computer Networks, 61:217 – 238. Specialissue on Future Internet Testbeds – Part I.

Santana, E. F. Z., Chaves, A. P., Gerosa, M. A., Kon, F., andMilojicic, D. S. (2016). Software platforms for smartcities: Concepts, requirements, challenges, and a uni-fied reference architecture. CoRR, abs/1609.08089.

Soldatos, J., Kefalakis, N., Hauswirth, M., Serrano, M.,Calbimonte, J.-P., Riahi, M., Aberer, K., Jayaraman,P. P., Zaslavsky, A., Zarko, I. P., Skorin-Kapov, L., andHerzog, R. (2015). OpenIoT: Open Source Internet-of-Things in the Cloud, pages 13–25. Springer Inter-national Publishing, Cham.

Taibi, F. (2013). Reusability of open-source program code:A conceptual model and empirical investigation. SIG-SOFT Softw. Eng. Notes, 38(4):1–5.

Villanueva, F. J., Santofimia, M. J., Barba, J., and Lopes,J. C. (2013). Civitas: The smart city middleware, fromsensors to big data. In Innovative Mobile and InternetServices in Ubiquitous Computing (IMIS), pages 445–450.