Full Terms & Conditions of access and use can be found athttp://www.tandfonline.com/action/journalInformation?journalCode=tcyb20

Cyber-Physical Systems

ISSN: 2333-5777 (Print) 2333-5785 (Online) Journal homepage: http://www.tandfonline.com/loi/tcyb20

MapTalk: mosaicking physical objects into thecyber world

Yi-Bing Lin, Min-Zheng Shieh, Yun-Wei Lin & Hsin-Ya Chen

To cite this article: Yi-Bing Lin, Min-Zheng Shieh, Yun-Wei Lin & Hsin-Ya Chen (2018):MapTalk: mosaicking physical objects into the cyber world, Cyber-Physical Systems, DOI:10.1080/23335777.2018.1510856

To link to this article: https://doi.org/10.1080/23335777.2018.1510856

Published online: 28 Aug 2018.

Submit your article to this journal

View Crossmark data

ARTICLE

MapTalk: mosaicking physical objects into the cyberworldYi-Bing Lin a, Min-Zheng Shiehb, Yun-Wei Lina and Hsin-Ya Chena

aDepartment of Computer Science, National Chiao Tung University, Hsinchu, Taiwan; bInformationTechnology Service Center, National Chiao Tung University, Hsinchu, Taiwan

ABSTRACTDigital map is considered as a cyber world, which maps visualrepresentations (cyber objects) to the physical objects in thereal world and allows the user to interact with these physicalobjects through their cyber representations. However, it typi-cally requires significant programming effort to create a mapapplication. This paper proposes MapTalk, a web-based visualmap platform that allows the user to interact with the physicalobjects through their cyber representations in a visual map.We show how the administrator can add applications to themap without any programming effort. The novel idea in ourapproach is to utilise the IoT concept. Specifically, we imple-ment the map as an output IoT device and all physical objectsto bemosaicked in themap as input IoT devices. We showhowto automatically create the device features of IoT devices whenthe administrator creates applications in the Map web page.We have deployed over 20 services in MapTalk including smarthome, smart farm, tracking (bus, person, dog, etc), video mon-itoring, travel route planning, sensing of CO2, PM2.5, Internetof Things (IoT). temperature, humidity, and so on.

Abbreviation: Internet of Things (IoT)

ARTICLE HISTORYReceived 3 August 2018Accepted 3 August 2018

KEYWORDSInternet of Things; location-based applications; sensors;Google Map

1. Introduction

Angelina Jolie said: ‘Anytime I feel lost, I pull out a map and stare. I stare until I havereminded myself that life is a giant adventure, so much to do, to see.’ Today, digitalmap is considered as a cyber world, which maps visual representations (cyberobjects) to the physical objects in the real world and allows the user to interactwith these physical objects through their cyber representations. Tools such asGoogle Maps offers the Application Programming Interface (API) to support inter-action between the services/applications and the map for visual illustration.Significant amounts of location information are already built in the map tools,which can be conveniently reused by various applications.

CONTACT Yi-Bing Lin liny@csie.nctu.edu.tw National Chiao Tung University, Hsinchu 300, Taiwan

CYBER-PHYSICAL SYSTEMShttps://doi.org/10.1080/23335777.2018.1510856

© 2018 Informa UK Limited, trading as Taylor & Francis Group

In Indonesia, the digital land certificate mapping application was developed [1]to assist the public and land officers to identify the land locations. The coordinatesare determined by GPS. As the author reported, the precision of this application isnot as good as expected and is in the enhancement process. The work in [2]developed a travel-planning platform based on the Google Maps, which utilisesTraNews and data sources to assist planning itineraries to enhance user travelexperience. In [3], the authors describe how to integrate the .NET technologywith the Google Maps API interface to develop a web portal for driver applications.To add user-friendly interactive applications on websites based on Google Maps, aninstant messaging system designed by Flex is described in [4]. In this study, a travelapplication embedded in a micro-blog system was implemented to displaydynamic map content. The aforementioned studies demonstrated the power ofmap tools and the values of location applications. However, developing newapplications in these approaches requires intensive programming efforts. To resolvethis issue, we propose MapTalk that allows the user to conveniently developapplications on digital map without programing effort. The novel idea of MapTalkis to utilise the IoT concept. Specifically, we implement the map as an output IoTdevice and all physical objects to be mosaicked in the map as output IoT devices.We use the term ‘designer’ to describe the person who develops new features ofMapTalk and is required the programming skills for Google Maps API, Flask,SQLAlchemy, WebSocket, Python, JavaScript, jQuery, HTML and CSS. An ‘adminis-trator’ is the personwho builds the location applications usingMapTalk GUIwithoutextra programming effort. This paper is organised as follows. Section 2 describes thecyber representation of a physical object. Section 3 shows how to create anapplication and its cyber objects in MapTalk. Section 4 elaborates on how tomosaicphysical objects into MapTalk. Section 5 describes how the visual map is imple-mented as an IoT device in MapTalk.

2. Cyber representations of physical objects in MapTalk

MapTalk is a simple version of Integrated Operation Centre (IOC) software that allowsefficient and convenient administration of intelligent environments such as smartcampus [5]. In National Chiao Tung University (NCTU), the physical objects (schoolbuses, parking lot sensors, washing machines, dryers, curtains, air conditioners,environmental sensors) are managed by the MapTalk IOC (a web page illustratedin Figure 1) that allows the university technical staff to conveniently administrate allsmart campus applications under one web-based graphical user interface (GUI).

MapTalk effectively mosaics physical objects of various IoT applications in a2D map. Our approach can automatically accommodate IoT applications withfew simple steps. The web-based GUI of MapTalk is considered as a cyberworld that interacts with the physical objects and visually shows them to theviewer (a user) of these applications. Therefore, MapTalk allows the user to talkand listen to physical objects through their cyber counterparts in the map.

2 Y.-B. LIN ET AL.

An IoT application is represented as a colored button for a drop-down list(Figure 1(a)). When the button is clicked, the names of all physical objects ofthat application are shown in the list. The user can select one or all objects.Then the cyber objects (visual representations of the physical objects) areshown in the map according to their physical locations.

Figure 2 illustrates the structure of a cyber object with the followingattributes:

Layout (Figure 2(a)): A cyber object can be a picture object (e.g. a camera inFigure 1(b)) or a geometric shape object. A geometric shape object can be arounded rectangle (Figure 3(a)), a circle (Figure 3b)) or a rounded square (Figure 3(c)), which is created with following features.

Figure 1. A 2D visual map based on MapTalk.

CYBER-PHYSICAL SYSTEMS 3

● Name (Figure 2(2)): In the space surrounded by the geometric shape, thedesigner can use a letter to specify the abbreviated object name; for exam-ple, ‘W’ for the washing machines in a laundry room (Figure 3(1)) and ‘P’ forPM2.5 (Figure 3(2)). The shape of an object with abbreviated letter is eitherrounded rectangle (Figure 3(a)) or rounded square (Figure 3(c)). On the otherhand, an object without the abbreviated letter is a circle (Figure 3(b)).

● Colour (Figure 2(3)): The background colour of a rounded rectangularor a circle object is the same as that of its application drop-down menubutton. Different applications are automatically assigned different col-ors. For example, the colour for washing machines in a laundry room issky blue (Figure 3(3)), and the colour for a tracked bus is violet red(Figure 3(4)).

● Value (Figure 2(4)): A text value (Figure 2(5)) can be shown inside arounded rectangular or a circle object. For example, the value 9(Figure 3(5)) represents the number of idle washing machines, and thevalue 2 (Figure 3 (6)) is the identity of a tracked bus. The colour (Figure 2

Cyber Object

Layout

PictureGeometric

Shape

Name Color Value

Text Color

Min Max

Location

Stationary Movable

Trajectory

Description

History Action

1

2 43

5 6

7

8 9

10

11

12 13

Figure 2. The structure of cyber objects in MapTalk.

(a) Round rectangle (b) circle

W: 9 2 P261 5 3 4

(c) rounded square

Figure 3. Layouts for geographic shape objects Round rectangle (b) circle (c) rounded square.

4 Y.-B. LIN ET AL.

(6)) fills the inside space of a rounded square object represents the stateof the object. The colours are mapped to a range [Min, Max] of floatnumbers. For example, in the PM2.5 application, the PM2.5 value range[1,10] is mapped to colour blue, [11,20] to green, [21, 30] to yellow, [31,40]to orange, and any value larger than 41 is mapped to red. The colourspectrum is shown in Figure 1(e).

● Location (Figure 2 (7)): A non-tracked object is stationary (Figure 2(8)), and itslocation is fixed in the map. A tracked object is movable (Figure 2 (9)), and itslocation is dynamically updated in the map. The tracked objects are used intracking people, animal, cars, and so on. The trajectory of a tracked object(Figure 2(10)) can be shownwith the same colour as the letter in the object (e.g.the trajectory and object letter 2 has the colour orange; see Figure 1(c) and (g)).

Description (Figure 2 (11)): When the user clicks the left button of themouse on a cyber object, a dialog box pops up (Figure 1(d)), which typicallycontains a text to describe the object or a hyperlink leading to a historicalchart (Figure 2 (12)) or action buttons (Figure 2 (13)) allowing the user tointeract with the physical objects.

The cyber objects can also be classified into two categories: single-attributeand composite-attribute. A single-attribute object represents a set of homoge-neous sensors. A composite-attribute object represents a set consisting of hetero-geneous sensors, controllers and/or actuators (to be elaborated in Section 4).

The menu bar (Figure 1(h)) consists of three types of buttons. All applicationsmanaged in MapTalk can be found in the App drop-down menu (Figure 1(i)). Therouting button (Figure 1(j)) provides travel route planning by allowing the user tospecifying the start and destination locations. This application is the same as thatoffered in Google Maps, and the details are given in Section 5. The menu bar alsoprovides a coloured button for a frequently accessed application (Figure 1(a)).

3. Creating a location application in MapTalk

The administrator creates and manages the physical objects of a location applica-tion through the MapTalk administration web page. This web page is the same asthat viewed by the user except that in the App drop-down menu (Figure 1(i)),there is the ‘Add app’ item (see Figure 7(a)). When the administrator clicks on thisitem, the application setup bar is shown below the menu bar (Figure 4(a)-(l)). Thisbar consists of eight fields. The Name fields (Figure 4(a)) allow specification of theapplication name (e.g. WashingMachine). The Location field (Figure 4(b)) imple-ments the location attribute of the cyber object (Figure 2(7)), where either‘Stationary’ or ‘Movable’ should be selected.

The Layout field (Figure 4(c)) implements the layout attribute (Figure 2(1)),which provides three options. If the Picture item (Figure 4(d)) is selected, the

CYBER-PHYSICAL SYSTEMS 5

window pops up a dialog box that allows the administrator to fill the Picture URL(Figure 5) linking to the picture to be shown in the MapTalk GUI. If the ‘No Letter’item (geometric shape without abbreviated name; see Figure 4(e)) is selected, thegeometric shape will be circle, and no abbreviated letter will be shown inside thecircle. If a letter field (geometric shape with abbreviated name; see Figure 4(f)) isselected, the geometric shape will be rounded rectangle or rounded square, andthe selected letter will be shown inside the shape.

The Value field (Figure 4(g)) implements the value attribute of the cyber object(Figure 2(4)) with two options. If the Text item (Figure 4(h)) is selected, the shape ofthe cyber object will be rounded rectangle. If the Colour field (Figure 4(i)) is selected,a dialog box (Figure 6) pops up to fill theminimumand themaximumvalues for thecolour range. In this case, the shape of the cyber object is rounded square.

The Quick Access field (Figure 4(j)) allows the administrator to put thecreated application in the menu bar (e.g. Figure 7(c)) for quick access. Afterthe application is created, the administrator can save the setup (Figure 4(k)) orremove the setup (Figure 4(l)). The created application is always included as anitem in the App drop-down menu (Figure 7(b)).

To add a new object in the application (e.g. WashingMachine), the admin-istrator clicks the Add WashingMachine item (Figure 7(d)), and the object setupbar is shown below the menu bar (see Figure 8 (a-c, e, f)). This bar consists ofsix fields.

The Name field (Figure 8(a)) specifies the object name. The value of theLocation field (Figure 8(b)) can be obtained by either manually inputting thelatitude and the longitude (Figure 8(c)) or clicking the mouse at the location ofthe map. After this field is filled, the cyber object is shown in the map(Figure 8(d)).

Figure 4. Application creation and setup.

6 Y.-B. LIN ET AL.

The Description field (Figure 8(e)) implements the Description attribute of thecyber object (Figure 2(11)), which allows specifying the text description or thehyperlink to a history web site (to be elaborated in the next section). After thecyber object is set up, the administrator clicks the ‘Save’ button (Figure 8(f)) toadd this cyber object to the application.

The washing machine example illustrates the creation of a stationaryobject. For movable objects such as buses, the creation is basically thesame except for the following differences. The ‘Movable’ item is selectedfrom the Location field (Figure 4(b)). If the Picture or a Letter field isselected from the Layout field (Figure 4(c)), all buses will have the sameappearance in the map. If it is desirable to distinguish the tracked buses,‘No Letter’ item is selected. In this setup, the geometric shape will be circle.No abbreviated name will be shown inside the circle; instead, the bus

Figure 5. Picture URL dialog box.

Figure 6. Colour range dialog box.

Figure 7. Application creation and setup.

CYBER-PHYSICAL SYSTEMS 7

identity is shown. We do not need to create individual bus objects as wedid for washing machines in Figure 8. When a new bus joins in thisapplication, its location and identity (an integer) are sent to MapTalk (tobe described in the next section) and automatically shown in the map (e.g.Figure 1(f,g)).

4. Mosaicking physical objects into MapTalk

MapTalk is an enhancement of IoTtalk, an IoT application platform for IoTdevice interaction and management [6–9]. In this platform, an input IoTdevice sends data to the IoTtalk server for manipulation, and then theresults are sent to one or more output IoT devices to drive these devices.MapTalk develops a visual map as an output IoT device called ‘Map’. Anyapplication of the physical objects to be mosaicked in the map is imple-mented as an input IoT device in MapTalk. For example, a laundry roomwith 20 washing machines and 10 dryers is implemented as two single-attribute input devices called LdryRmWM and LdryRmD, respectively. Theadministrator manipulates these devices through the MapTalk GUI. MapTalkinteracts with the physical objects of an application based on the conceptcalled ‘device model’. Through the procedure described in [10], two devicemodels for laundry room are built in MapTalk and are included in the devicemodel drop-down list in the menu bar of the MapTalk GUI (Figure 9(1)).When the designer selects ‘Washing Machine’ from the list, the icon for theLdryRmWM device is shown in the left-hand side of the graphical window(Figure 9(2)). The physical objects of the same type are grouped andrepresented as a single-attribute cyber object in MapTalk through the con-cept of ‘device feature’ (DF). The DFs of an input device are called input DFs(IDFs). Therefore, the 20 washing machines in the laundry room are repre-sented by the IDF icon WashMach-I inside LdryRmWM (Figure 9(3)). Similarly,

Figure 8. Cyber object creation and setup.

8 Y.-B. LIN ET AL.

the 10 dryers in the laundry room are represented by the IDF icon Dryer-Iinside LdryRmD (Figure 9(4)). We note that all IDF names are appended with‘-I’ and all ODF names are appended with ‘-O’. To identify the location at themap, both LdryRmWM and LdryRmD also include an IDF called Location-I.This IDF may represent a real location sensor (e.g. the GPS receiver oriBeacon) or a preset latitude and longitude value pair.

As an output IoT device, the Map icon is placed in the right-hand side of thegraphical window (Figure 9(5)). From the above description, we treat a devicemodel as a collection of DFs. An input (output) device is a set of IDFs (ODFs).The MapTalk server links an IDF to an ODF by two line segments connected toa small ‘Join’ circle. By clicking the circle, the designer can implement thefunctions to manipulate the data sent from the IDF to the ODF. In Figure 9,WashMach-I of LdryRmWM connects to Map or more appropriately, connectsto the WashMach-O ODF of a cyber IoT device called WMStatus (Figure 9(6)).The location of the washing machines is also passed to WMStatus through theLocWM-I (Figure 9(7)) and LocWM-O (Figure 9(8)) connected by Join 1.Similarly, Dryer-I is connected to the IoT device DryerStatus (Figure 9(9))through Join 3. If an IoT device includes both IDFs and ODFs, then it isrepresented by an input device and an output device in MapTalk. WMStatushas both input device and output device (Figure 9(6) and (10)). Similarly,DryerStatus is represented by both input and output devices.

Figure 9. Laundry application configured in the MapTalk GUI.

CYBER-PHYSICAL SYSTEMS 9

The output device of WMStatus lists the status of each washing machine andthe history line chart for that machine in a web page (Figure 10). The inputdevice of WMStatus counts the number of idle washing machines and sends thisnumber out through the IdleWM-I IDF (Figure 9(10)). This IDF is a counterassociated with the latitude and longitude pair obtained from LocWM-O. TheIdleWM-I IDF is connected to the IdleWM-O ODF of Map (Figure 9(11)) throughJoin 5, and the map shows the number of idle washing machines in the map(see Figure 1(d); the rounded rectangle indicates nine idle washing machines inthis example). The administrator fills the hyperlink for the WMStatus web pagein the Description field in the object setup bar in Figure 8(e). Therefore, whenthe user clicks the hyperlink in Figure 1(d), the web page in Figure 10 pops up.

Figure 9 also includes a BusTracking icon connected to Map through Join 7.The BusID-I IDF (Figure 9 (12)) provides the locations of buses been tracked(Figure 1(f, g)). As mentioned before, Figure 1 shows tracking of two busesrepresented by violet-red circles marked with the spring-green number 0 (thefirst bus) and the orange number 2 (the second bus) with the tails (colored linesegments) indicating the historic traces of the moving buses.

In the Map output device, the ODFs (i.e. BusID-O, IdleWM-O, IdleDryer-O) areautomatically created after the administrator has set up the applications (Bus,Washing Machine, and Dryer) in Map web page and clicks the ‘Save’ button(Figure 4(k)).

In Figure 9, the input devices connected directly or indirectly to Map aresingle-attribute cyber objects. To mosaic a composite-attribute cyber object toMap, there are two alternatives. Typically, a web page is implemented for amosaic composite-attribute input device. An example is RoomTalk, a smarthome application that consists of multiple switches and sensors to control thehome appliances. Figure 11 illustrates a part of the web page for RoomTalk.The right-hand side of the figure shows nine buttons for controlling curtains(there are more than 30 buttons in RoomTalk). The web page also mosaics avideo streaming platform in the left-hand side of Figure 11. The camera in thevideo streaming platform shows all actuators (ODFs) controlled by the IDFs ofRoomTalk.

The RoomTalk configuration in the IoTtalk GUI is illustrated in Figure 12. Inthis configuraiton, no RoomTalk IDFs are connected to Map. The administratoronly needs to create the RoomTalk application and the cyber object in the Mapweb pages (e.g. Figures 4 and 8), and there is no need to make any connectionin the MapTalk GUI (e.g. Figure 9).

The input devices for single-attribute cyber objects in Figure 9 can also beimplemented by the input device for a composite-attribute cyber object.Suppose that the washing machines and the dryers mentioned in Figure 9are located in the same laundry room. Then they can be implemented as acomposite-attribute cyber object by merging both LdryRmWM and LdryRmDtogether with a temperature sensor into an input device called LdryRm

10 Y.-B. LIN ET AL.

(Figure 13(1)). The location of LdryRm is specified at LocLR-I, and LdryRmWMand LdryRmD are represented by the WashMach-I and the Dryer-I IDFs, respec-tively. All IDFs are connected to an output device called LdryRmStatus(Figure 13 (2)) that is similar to WMStatus and DryerStatus. A web page isautomatically created for LdryRmStatus (similar to the one illustrated inFigure 10)

In the first alternative to mosaic a composite-attribute input device intoMap, no IDFs of the input device is connected to the ODFs of Map. Instead,the administrator simply inputs the hyperlinks of the RoomTalk and theLdryRmStatus web pages in the Description field (Figure 8(e)). In this alter-native, when the user clicks the RoomTalk (LdryRm) cyber object in themap, the RoomTalk (LdryRmStatus) web page hyperlink pops up. If the userclicks the hyperlink, the web page will be shown on the screen formanipulation.

Figure. 14 illustrates the relationship between LdryRm, RoomTalk and Map.In the first alternative, although all of these three IoT devices connected toMapTalk server (Figure. 14 (1), (2), and (3)), their IDFs are not connected to theMap ODFs. Instead, the hyperlinks of LdryRm and RoomTalk are mosaicked inthe map (Figure. 14 (4) and (5)).

Figure 10. The web page for WMStatus (Green: idle; Red: busy).

CYBER-PHYSICAL SYSTEMS 11

In the second alternative, some IDFs of a composite-attribute input deviceare connected to the ODFs of Map (see Join 5 in Figure 15). In this example,the administrator has created a temperature application, which results in theODF Temperature-O of Map in the MapTalk GUI (Figure 15(1)). When theadministrator connects Temperature-I of LdryRm (Figure 15(2)) and RoomTalk(Figure 15(3)) to Temperature-O of Map, the temperatures of the laundry roomand the smart home demo room will be directly shown in the visual mapthrough the Temperature application of Map. In this alternative, the physical

Figure 11. The web page for RoomTalk (partial view).

Figure 12. The (partial) RoomTalk configuration.

12 Y.-B. LIN ET AL.

and cyber temperature objects also interact through the MapTalk server (i.e.through (1), (2) and (6) in Figure. 14).

5. The map implementation as an IoT device

This section describes how Map is implemented as an IoT device. Dependingon the context, we will use the term ‘user’ (Figure 16(8)) to represent ‘admin-istrator’ and ‘user’ interchangeably. Figure 16 illustrates the functional blocksof the Map device. The MsgHandler (Figure 16(1)) is the core engine of theMap device, which is responsible for interaction with the MapTalk server(Figure 16(2)) through three tasks.

● The Register task (Figure 16(3)) connects the Map device to the MapTalkserver. The Map device interacts with the MapTalk through MQTT or HTTP.After registration, the user/administrator can manipulate the map asdescribed in Section 3.

● The Update task (Figure 16(4)) informs the MapTalk server of the changefor Map. When the administrator adds a new application (e.g. WashingMachine) to Map, the MapTalk server reflects such changes of Map icon inFigure 9 by automatically showing the IdleWM-O ODF.

● In the Pull task (Figure 16(5)) the MsgHandler receives the data of thecyber objects (IDFs) from the MapTalk server through the Map’s ODFs. TheMsgHandler then asks the DB Control (Figure 16(6)) to store the ODF data(with the format [lat, lng, name, value, timestamp]) in the database MapDB (Figure 16(7)). The protocol exercised between the MsgHandler andthe DB Control is HTTP.

● After an input device (e.g. LdryRmWM) has been connected to Mapdescribed in Section 4, the IDFs of the input device send the data to

Figure 13. LdryRm and LdryRmStatus configuration.

CYBER-PHYSICAL SYSTEMS 13

Map’s ODFs through the Join connections. The MsgHandler retrieves thedata through the Pull task described above. Also, the MsgHandlerinstructs the Map Control (Figure 16(10)) to update the values of thecorresponding cyber objects (Figure 2(4)) and then instructs the Graphic

Figure. 14. The relationship between LdryRm, RoomTalk and Map.

Figure 15. The second alternative to mosaic a composite-attribute input device into Map.

14 Y.-B. LIN ET AL.

Render (Figure 16(11)) to show the results to the user following themessage path (2) ! (5)→(1)→(1) ! (10) ! (11)→(8) in Figure 16.

● The DB Control is implemented by Flask, which interacts with the Map DBthrough ORM API. Besides storing the ODF data, the Map DB also storesthe app buttons (see Figure 1(a)) with the format [name, location, layout,value, quick access], the setups for every stationary cyber object (seeFigure 3(a,c)) with the format [name, location, description], and the setupsfor every movable cyber object (see Figure 3(b)) with the format [name,description].

● When the user manipulates the application through the Map web page,the User Event Handler (Figure 16(9)) catches the events and dispatchesthese events to other components of the Map device.

● When the administrator adds a new application (e.g. washing machines) byselecting the ‘Add app’ item (Figure 7(a)), the User Event Handler instructsthe Graphic Render to generate the application setup bar (Figure 4(a)-(l))through the message path (8)(9) ! (11) ! (8) in Figure 16).

● When the administrator presses the ‘Save’ button (Figure 4(k)), the User EventHandler sends the app setup information to the MsgHandler. The MsgHandlerinvokes the Update task to the MapTalk server to create related ODF (i.e.IdleWM-O) for the Washing Machine (message path (1)→(4) ! →(2)). TheMsgHandler also issues an HTTP request that instructs the DB Control to savethe setup information to the Map DB (message path (1)→(6)→(7)).

● After creation of an application, the administrator can add a cyber object tothe application by pressing, for example, the ‘Add WashingMachine’ item(Figure 7(d)). The User Event Handler requests the Graphic Render to showthe object setup bar (Figure 8(a-f)). When the administrator clicks a locationin the map, the User Event Handler instructs the Map Control to use theGoogle Maps JavaScript API of the Google Maps Platform (Figure 16(14)) toobtain the latitude and the longitude of the hit position (message path

HTTP

DB

ControlUpdateRegisterPull

MapTalk Server

ORM

APIMap

DB

User Event

Handler

Graphic Render

RTSP

Map

Control

Video

Control

Google

Maps

Platform

Streaming

Server

User

MsgHandler

HTTP 5 3 4

1

9

6

7

13

14

2

8

10

1112

Figure 16. The functional blocks of the Map device.

CYBER-PHYSICAL SYSTEMS 15

(8) ! (9) ! (10)→(14)→(10)), and create the cyber object to be sent to theGraphic Render for display (message path (10) ! (11) ! (8)).

● When the administrator presses the ‘Save’ button in Figure 8(f), the UserEvent Handler dispatches the event to the MsgHandler. The MsgHandlersends an HTTP request to the DB Control to save the new object in theMap DB and retrieves the information of the application and the cyberobjects (message path (1) ! (6) ! (7) ! (6) ! (1)). The MsgHandler thensends them to the Graphic Render to generate the application quickaccess drop-down list button in the menu bar and all cyber object itemsof the drop-down list in the Map web page (message path (1)→(11) !(8)). In parallel, the User Event Handler instructs the Map Control to sendan HTTP request to the Google Maps Platform to obtain the JavaScript APIfor future usage (message path (9) ! (10) ! (14) ! (10)).

● When the user selects the ‘show all’ item or a cyber object item in anapplication drop-down menu button (Figure 1(a)), the User Event Handlerdispatches the event to the Map Control. The Map Control uses the GoogleMaps JavaScript API to layout all cyber objects (or a specific object) andshows them in the map (message path (8) ! (9) ! (10) ! (11) ! (8)).

● If the user clicks the Routing button (Figure 1(j)), the event is dispatchedto the Map Control. The Map Control instructs the Graphic Render toshow a pop up window (Figure 17(a)) that allows the user to specify thestarting point and the destination (message path (8) ! (9) ! (10) !(11)→(8)). After the user has set up the starting point/destination and

Figure 17. Path routing.

16 Y.-B. LIN ET AL.

clicks the ‘Start Routing’ button (Figure 17(b)), the User Event Handlerdispatches the event to the Map Control. The Map Control invokes thedirections service of the Google Maps Platform to obtain the suggestedtravel path, and draw it through the JavaScript API. The result is sent tothe Graphic Render to show in the map (Figure 17(c)). The message pathis (8)→(9)→(10)→(14)→(10)→(11)→(8).

● When the user clicks on a camera cyber object (Figure 1(b)), the User EventHandler dispatches the event to the Map Control. The Map Control orientsthe center of the map above the camera cyber object and sends thestreaming request to the Video Control (Figure 16(12)). The message pathis (8)→(9)→(10)→(12). The Video Control then sends the correspondingRTSP request to the Streaming Server (Figure 16(13)), and the video isviewed by the user through the message path ((13)→(12)→(11)→(8)). Theuser watches the video in the map as illustrated in Figure 18.

6. Conclusions

This paper proposed MapTalk, a web-based visual map platform that enablesthe user to interact with the physical objects through their cyber representa-tions in a visual map. To allow the administrator to add applications to themap without any programming effort, the implementation of MapTalk is nottrivial. The novel idea in our approach is to utilise the IoT concept. Specifically,based on IoTtalk, we implement the map as an output IoT device and allphysical objects to be mosaicked in the map as input IoT devices. We showedhow to automatically create the device features of an IoT device when the

Figure 18. Video streaming in MapTalk.

CYBER-PHYSICAL SYSTEMS 17

administrator creates the corresponding application in the Map web page. Theadvantages of MapTalk include:

● Cyber physical interaction of visual map is achieved through a well-established IoT platform.

● Any physical system (such as school buses, laundry rooms, etc.) is auto-matically represented as a cyber IoT device and then easily accommo-dated in the map for illustration.

● No programming effort is required to add a new application in the map.

In NCTU MapTalk IOC, we have included over 20 services including smarthome, smart farm, tracking (bus, person, dog, etc), video monitoring, travelroute planning, sensing of CO2, PM2.5, temperature, humidity, and so on.

Disclosure statement

No potential conflict of interest was reported by the authors.

Funding

This work was supported by the Ministry of Education [the SPROUT Project-Center for OpenIntelligent Connectivity of NCTU]; Ministry of Science and Technology of the Republic ofChina [Grant 106-2221-E-009 −006 and Grant 106-2221- E-0];Academia Sinica [AS-105-TP-A07].

ORCID

Yi-Bing Lin http://orcid.org/0000-0001-6841-4718

References

[1] Lin Y-B, Chen L-K, Shieh M-Z, et al. CampusTalk: IoT devices and their interestingfeatures on campus applications. IEEE Access. 2018b December;6(1):26036–26 046.

[2] Windarni VA, Sediyono E, Setiawan A. Using GPS and Google maps for mappingdigital land certificates, 2016 International Conference on Informatics and Computing(ICIC), Mataram, 2016, pp. 422-426. doi: 10.1109/IAC.2016.7905756

[3] Chung-Hua C, Chenyang, H. Platform for travel planning by using google maps. In:16th IEEE International Conference on mobile data management. 2015.16th IEEEInternational Conference on Mobile Data Management, Pittsburgh, PA, 2015, p. 120-125. doi:10.1109/MDM.2015.13

[4] Konarski M, Zabierowski W. Using Google Maps API along with technology .NET.International Conference on modern problems of radio engineering.Telecommunications Comput Sci (TCSET). 2010. Lviv-Slavske, 2010, p. 180–182.

[5] Cao Y, Xu ZH, Yan ZX, et al. An instant messaging system based on Google Map. In:2nd International Conference on advanced computer control. 2010.

18 Y.-B. LIN ET AL.

[6] Lin Y.-B., Lin YW, Chih CY, et al. EasyConnect: a management system for IoT devicesand its applications for interactive design and art. IEEE Internet Things J. 2015December;2(6):551–561.

[7] Lin Y.-B., Lin YW, Huang CM, et al. IoTtalk: A Management Platform for ReconfigurableSensor Devices," in IEEE Internet of Things Journal, vol. 4, no. 5, pp. 1552-1562, Oct.2017. doi:10.1109/JIOT.2017.2682100.

[8] Lin Y.-B., Lin YW, Hsiao CY, et al. 2017b June Location-based IoT applications oncampus: the IoTtalk approach, Pervasive and mobile computing Vol. 40, 660–673.

[9] Lin Y.-W., Lin YB, Hsiao CY, et al.IoTtalk-RC: Sensors AsUniversal Remote Control forAftermarket Home Appliances," in IEEE Internet of Things Journal, vol. 4, no. 4, pp.1104-1112, Aug.2017. doi: 10.1109/JIOT.2017.2715859.

[10] Lin Y.-W., Lin YB, Yang MT, et al. ArduTalk: an Arduino network application develop-ment platform based on IoTtalk. IEEE Syst J. 2018a. accepted and to appear

CYBER-PHYSICAL SYSTEMS 19