web 2.0 and estate maps

8/8/2019 Web 2.0 and Estate Maps

1/87

Web 2.0 and estateMaps

Final Year Project

BEng Software Engineering with Industrial Experience

The University of Manchester

Author: Imran Sarwar

Supervisor: Sean Bechhofer

April, 2008


2/87

Imran Sarwar 2 SEwI.E Final year Project

Disclaimer

No portion of the work referred to in the dissertation has been submitted in support of an application

for another degree or qualification of this or any other university or other institution of learning.


3/87


Abstract

The World Wide Web (WWW) is a well known phenomenon which enables billions of users to

connect to an information pool used and updated by users across the globe. Closely linked with the

WWW is a fairly new and loosely defined concept called Web 2.0 which is defined as the second

generation of the WWW.

This report details the development of my hybrid application which exploits a combination of freely

available online services, to create a lightweight mashup application. The application also provides

information concerning the surrounding area of the property, creating a rich and informative

experience for the end user.

Included in this report are the project requirements, design, implementation, testing and the

background surrounding Web 2.0 and the different technologies which are associated with it.


4/87


Web 2.0 is a set of economic, social, and technology trends that collectively form the basis for

the next generation of the Interneta more mature ,distinctive medium characterized by user

participation, openness, and network effects.

~ Tim O'Reilly


5/87


Acknowledgements

I would like to thank my supervisor Sean Bechhofer for his guidance and direction throughout the

project. I would also like to thank my parents and family members for their support and advice during

the project.

I would like to say special thanks to my friends Asif, Michael and Shehzad for their support during the

project and keeping me in focus during the final year.


6/87


Table of ContentsList of figures.........................................................................................................................................9

1. Introduction..................................................................................................................................11

1.1 Aims and Objectives ................................................................................................................ 11

1.2 Deliverables ............................................................................................................................. 12

1.3 Report Structure ....................................................................................................................... 12

2. Background ..................................................................................................................................13

2.1 Existing services ...................................................................................................................... 13

2.2 Putting Web 2.0 into context.................................................................................................... 14

2.3 Related Technologies............................................................................................................... 15

2.3.1 Web Services ..................................................................................................................... 15

2.3.2 AJAX ................................................................................................................................. 17

2.3.3 Web feeds.......................................................................................................................... 19

2.3.4 XML.................................................................................................................................. 20

2.3.5 Mashup.............................................................................................................................. 20

2.3.6 RSS and GeoRSS............................................................................................................... 22

2.3.7 XSLT.................................................................................................................................. 23

2.3.7 Google Maps ...................................................................................................................... 24

2.3.8 Nestoria .............................................................................................................................. 24

2.3.9 Flickr .................................................................................................................................. 25

2.3.10 PlanningAlerts................................................................................................................... 253. Design............................................................................................................................................27

3.1 Requirements ........................................................................................................................... 27

3.1.1 Functional requirements..................................................................................................... 27

3.1.2 Non-functional Requirements ........................................................................................... 28

3.2 Design Methodology................................................................................................................ 28

3.3 System Architecture................................................................................................................. 303.3.1 Presentation logic............................................................................................................... 30


7/87


3.3.2 Processing Logic ................................................................................................................ 32

3.3.3 Data Logic.......................................................................................................................... 32

3.3.4 Architecture design decisions ............................................................................................ 32

3.4 Problems .................................................................................................................................. 33

3.4.1 JavaScript security restrictions........................................................................................... 33

3.4.2 Location and crime statistics data ..................................................................................... 34

4. Implementation ............................................................................................................................36

4.1 Programming Language - The Microsoft vs. Java syndrome .................................................. 36

4.2 Development Environment ....................................................................................................... 36

4.3 Development tools .................................................................................................................... 37

4.4 Server Side Implementation...................................................................................................... 37

4.5 Server Code exploration ........................................................................................................... 40

4.5.1 Reading HTTP Requests.................................................................................................... 40

4.5.2 Processing a search request................................................................................................ 40

4.5.3 Processing a Graph Request............................................................................................... 45

4.5.4 Locating Flickr images....................................................................................................... 47

4.4.6 Processing the search bar ................................................................................................... 47

4.5 Client side code......................................................................................................................... 48

4.5.1 Google Maps instillation................................................................................................... 48

4.5.2 AJAX Requests and Response Caching............................................................................ 49

4.5.3 Plotting houses on the Google Map ................................................................................. 50

Predictive search bar ..................................................................................................................... 53

5. Testing...........................................................................................................................................54

5.1 Debugging................................................................................................................................. 54

5.2 White Box Testing .................................................................................................................... 54

5.3 Black box Testing ..................................................................................................................... 55

6. The Result .....................................................................................................................................57

6.1 A typical search request ............................................................................................................. 57


8/87


7. Conclusion ....................................................................................................................................62

7.1 Evaluation of the Implementation............................................................................................. 62

7.2 The future and extra functionality............................................................................................ 63

7.3 My Learning.............................................................................................................................. 65

8. Appendices....................................................................................................................................66

Appendix A - Development Plan ...................................................................................................... 67

Appendix B - Initial Requirements ................................................................................................... 68

Appendix C - XSLT file used to transform a house search request.................................................. 70

Appendix D - A typical XML file generated by the XSLT engine................................................... 71

Appendix E - A typical XML file for a house search request which is returned to the client .......... 75

Appendix F Mock-up interfaces .................................................................................................... 81

Mock-up 1..................................................................................................................................... 81

Mock-up 2..................................................................................................................................... 82

Appendix G Zillow test class ......................................................................................................... 83

References............................................................................................................................................84


9/87


List of figures

FIGURE 1 - Trends in mashups [1].......................................................................................................13

FIGURE 2 Nestoria interface [11]......................................................................................................14

FIGURE 3 - Web services architecture [4] .........................................................................................16

FIGURE 4 - Simple REST example. ...................................................................................................17

FIGURE 5 - The traditional model vs. the AJAX approach [5]. ..................................................18

FIGURE 6 - XML syntax Example [20]..............................................................................................20

FIGURE 7 - Mashup programming inputs and outputs [12]................................................................21

FIGURE 8 - RSS 2.0 specification [13].................................................................................................22

FIGURE 9 - GeoRSS specification [14]................................................................................................23

FIGURE 10 - Template processor [16]..................................................................................................23

FIGURE 11 - XSL code to convert GeoRSS to XML...........................................................................24

FIGURE 12 - Iterative design approach [26].........................................................................................29

FIGURE 13 - EstateMaps architecture overview. .................................................................................30

FIGURE 14 - Predictive search bar. ......................................................................................................31

FIGURE 15 - Possible client side architecture. .....................................................................................33

FIGURE 16 - Accessing external services.............................................................................................34

FIGURE 17 - EstateMaps database structure.........................................................................................35

FIGURE 18 - UML class interaction diagram.......................................................................................38

FIGURE 19 - Reading a transformed house XML file..........................................................................42

FIGURE 20 - Process graph for reading a transformed house XML file...............................................43

FIGURE 21 - Double Array graph structure..........................................................................................46

FIGURE 22 - Process graph for the client side caching.........................................................................49


10/87


FIGURE 23 - Use case for plotting items on the Google map...............................................................51

FIGURE 24 - Tabbed Google Bubble....................................................................................................52

FIGURE 25 - Control flow graph for processing a House XML file on the client side........................55

FIGURE 26 - Black box testing results..................................................................................................56

FIGURE 27 - EstateMaps initial GUI....................................................................................................57

FIGURE 28 - EstateMaps loading screen..............................................................................................58

FIGURE 29 - House information displayed on the estateMaps GUI.....................................................58

FIGURE 30 - Crime statistics displayed on the estateMaps GUI..........................................................59

FIGURE 31 Flickr images displayed on the estateMaps GUI............................................................59

FIGURE 32 Searching the Google map..............................................................................................60

FIGURE 33 Displaying planning alerts on the estateMaps GUI........................................................61

FIGURE 34 Average house prices on the estateMaps GUI................................................................61


11/87


1. Introduction

The World Wide Web (WWW) is a fast and ever expanding market enabling users to communicate

and collaborate ideas on a global scale. A new and loosely defined concept has recently become the de

facto standard in creating web applications and websites. This new era of the internet is termed as

Web 2.0 where the user is a potential stakeholder in the website which was first conceptualised with

the increased trends in social networking and information sharing.

Many organisations such as Microsoft, Yahoo and Google have adopted the Web 2.0 principle by

making their data and services freely available to the public. These services are accessed via APIs

(Application Programming Interface) using a combination of technologies such as AJAX and XML

which are discussed in sections 2.3.2 and 2.3.4 respectively.

This report details the development of my hybrid/mashup application called estateMaps which uses a

number of freely available online services to create a rich and informative experience for the end user.

The report also describes the basic background and concepts of Web 2.0 and the technologies which

are associated with estateMaps such as REST, SOAP, XML and AJAX.

1.1 Aims and Objectives

The objectives of my project are to exploit the concepts of Web 2.0 and create an application which is

able to display saleable real estate geographically on a world map. The first objective identified for

estateMaps was to understand the different APIs and how data would have to be manipulated and

extracted from the different information sources. Many of the third party services use XML as their

primary language to manipulate and dispatch data when a request is generated from the service. This

required research into the structure of XML and GeoRSS documents which are discussed in sections

2.3.4 and 2.3.6 respectively.

The second objective was to join the different information sources into one single lightweight

application which as mentioned is termed a hybrid/mashup application. This required further research

into the different technologies available such as JavaScript and AJAX which have both been discussed

in the Background section of this report.

The high level objectives can be seen as;

Develop a Java servlet which can explicitly call third party services and generate response

based on the users request.

Create an application which is loosely coupled and easily upgradable.


12/87


Investigate the concepts of Web 2.0 and the technologies which surround this subject.

Develop an application which meets the requirements defined in section 3.1.

1.2 DeliverablesThe developed artefact will provide users with the ability to search for saleable real estate across the

UK with information about the surrounding area. This information will include crime statistics about

the region, average house prices, planning requests in the area and images relevant to the location.

The application will be formed as a fully operational web based application allowing the user to

navigate the features in a simplistic and familiar environment. Real estate will be displayed on a

geographical map with markers to represent the location of each property.

1.3 Report Structure

The report consists of several distinct chapters, each chapter builds upon the last which is geared

towards the implementation and evaluation of the project.

Chapter 2 discuses the background of some of the technologies which are used within the context of

Web 2.0 and are used in the implementation of the project.

The Design and the implementation of the application are discussed in Chapter 3 and 4. Chapter 3

describes the challenges associated with estateMaps and the architectural design. Chapter 4 describesthe implementation of the application and how the core components of the application are

implemented into the system.

Chapter 5 tests the system against the initial system requirements and other various testing techniques

such as Black and White Box testing. Chapter 6 ends the implementation and aims to evaluate the

effectiveness of my application against other products available on the market.

Finally Chapter 7 outlines the future of Web 2.0 and the possibilities of Web 3.0 already on the

horizon. Chapter 7 also outlines the direction and manifestation which the internet is taking.


13/87


2. BackgroundThis chapter is split into three distinct sections. The first section takes a brief look at some of the

existing services which are already available and how estateMaps differentiates from them. The

second section defines this idea of web 2.0 and how it affects the WWW in its current state. Finally

the last section lays out the technologies which are linked to the application domain of Web 2.0 and

those which are potentially used during the implementation phase of the project.

2.1 Existing services

With the increased trends in Web 2.0, there are thousands of existing mashups and many more being

created at an exceptional rate to create a new experience for the end user. Figure 1 shows the vogue in

mashup programming, these statistics are provided from ProgrammableWeb [1] a site which lists thelatest mashups and web services.

Figure 1 Trends in mashups [1].

Looking at the statistics shown in figure 1, there seems to be an increased interest in using mapping

services at this moment in time. Real estate is a relatively niche market and has not be been exploited

to its full potential. Combining real estate and a mapping service seems be a good basis for mashup

service hence the idea of estateMaps was formed.

There are many services which are able to list real estate geographically on a mapping service one of

which is the Nestoria [11] web service. Nestoria is of particular interest to estateMaps because it is the

main data driving force for the application. The Nestoria web service is mentioned in greater detail

during the third section of this chapter. Figure 2 shows the interface of the Nestoria web application it

provides the user with the key functionality of estateMaps but lacks information such as information

relevant to the search location and recent planning requests in the area.


14/87


Figure 2 Nestoria interface [11].

I also researched a number of real estate agents to see if they could offer a service which was as rich

and interactive as estateMaps, but to no prevail. Main&Main [10] which is a popular real estate agent

in the UK offers a similar service but has less functionality than both Nestoria and estateMaps.

Main&Main do not provide any information in regards to the surrounding area, their primary goal is to

inform the user of the location of the property utilising a Google map.

No examples could be found at the time of writing this report which uses AJAX to dynamically update

the map when the users make a new search request based in the UK. This keeps in context with the

idea of Web 2.0 to create applications which is interactive for the user and feels as if though it is

running seamlessly on their local machine.

2.2 Putting Web 2.0 into context

Web 2.0 is perceived to be the second generation of the internet, which aims to get the end user

involved in the web experience. The idea of Web 2.0 is not to change the current infrastructure of the

WWW, but to change the way developers and users interact with WebPages.

There are no governing specifications which classify a Web 2.0 application instead a number of

publishers such as Tim OReilly president of O'Reilly Media have published their views on what

characteristics classify a Web 2.0 application.

OReilly published an interesting article titled What Is Web 2.0 [2]. In this article OReilly mentions

the seven key competences of web 2.0 which have been summarised below.


15/87


The Web As Platform - Using the browser to run rich and interactive applications just as

though they are running natively on the users machine.

Harnessing Collective Intelligence Allowing users to become potential stakeholders in the

website by enabling them to add and manipulate the data such as blogs and Wikipedia.

Data is the Next Intel Inside The data is the key driving force of the application not from

one source but from many.

End of the Software Release Cycle Using the Internet as base operating system to run

applications so users dont have to install or update the software. The person/company

providing the service will update the application which separates the concern from the user.

Lightweight Programming Models The program should adopt a loosely coupled and flexibleapproach which allows other components and even different applications to utilise its

functionality. Using lightweight programming models means we can reach more people to

enable data collection and a more intelligent web [2].

Software Above the Level of a Single Device The application should not be restricted to one

platform it should be able to take advantage of all technologies available today and in the

future.

Rich User Experiences Allowing the user to take control over the web site and providing theuser with an experience which is interesting, innovative and informative in order to create ICT

as effective tool.

Looking at these characteristics we can already identify that there are number of practical uses for

Web 2.0 applications such as forums, blogs and even price comparison websites amongst a host of

others. EstateMaps adheres to the ideas put forward by OReilly, but the key feature can be seen as

integrating data from multiple information sources.

2.3 Related Technologies

2.3.1 Web Services

Web services are at the focal point of the web 2.0 concept; they enable developers to create

Lightweight Programming Models which as mentioned is one of the fundamental concepts of web

2.0. The World Wide Web Consortium (W3C) [3] defines a web service as

....A Web service is a software system designed to support interoperable machine-

to-machine interaction over a network. It has an interface described in a machine-


16/87


processable format (specifically WSDL). Other systems interact with the Web service

in a manner prescribed by its description using SOAP-messages, typically conveyed

using HTTP with an XML serialization in conjunction with other Web-related

standards. [3]

In simple terms a web service can be described as self-contained and self-describing components that

can be published, discovered, and invoked across the network.

A web service is normally provided in the form of an API, an interface which allows a web browser or

server to interact with another application. The computational logic is usually calculated on the host

which is providing the web service which explicitly separates the two applications. The main

protocols or architectures used for web services are SOAP and REST.

SOAP (Simple Object Access Protocol) is a computer protocol used for exchanging XML basedmessages over the internet. SOAP allows computer programs to communicate on different operating

systems, different technologies and programming languages. WSDL (Web Service Definition

Language) is used to describe the methods and the location of a web service. [3].

Figure 3 - Web services architecture [4].

The typical SOAP architecture is shown in figure 4, it captures how data is transmitted from the

requester to the provider. The service requester sends a SOAP message in the XML format to the

provider which instantiates a WSDL binding (connection) which the requester can use to invoke the

methods or functions of the provider.

Representational State Transfer (REST) is not a protocol instead it is termed as an architectural style

[6]. It exploits the current HTTP /HTTPS protocol in order to send requests to a web server in the


17/87


form of a URL (Uniform Resource Locator). HTTP is the set of rules for exchanging files (text,

graphic images, sound, video, and other multimedia files) on the WWW [10].

Each element on the internet is seen a resource and the URL is the resource locator [5]. The

representation is the information that the requester is interested in. The resource or data is sent back from the web server in a number of different standards which include XML, HTML, GIF etc.

Figure 4 highlights a simple GET request using the basic principles of REST. The starting portion of

the URL is the location of the resource i.e. the server. The second portion of the URL is the name of

the resource which the user wishes to retrieve, in this case an XML file called Text. The server

processes the second portion of the URL in order to send back the resource to the user. Both sending

and receiving the request exploits the HTTP protocol as illustrated in the example.

Figure 4 Simple REST example.

REST is of particular importance to estateMaps because a number of the third party services used

within the application use this architecture.

2.3.2 AJAX

Asynchronous JavaScript and XML (AJAX) is a relatively new concept which is able to transform the

way static WebPages are updated and refreshed in an intuitive manner. Following the principles of

Web 2.0 AJAX enables developers to create a new experience for the end user, WebPages with a

seamless flow of data exchange [7].

AJAX is not a new Scripting language instead it manipulates existing standards to create faster, more

reliably and friendlier user applications. AJAX allows portions of a website to be updated without the

need to update the entire page. This creates the perception that the application is actually running on

the local machine rather than a service over the WWW.

AJAX uses JavaScript as the scripting language of choice to instantiate AJAX function calls. XML is

the standardised way to retrieve data using the AJAX libraries. Data does not necessarily have to be in

HTTP GET / www.example.com/text

HTTP Response / XML

Resource

Resource

Location
http://www.example.com/texthttp://www.example.com/text


18/87


the XML format other formats are supported such as HTML (Hyper Text Mark-up Language, JSON

(JavaScript Object Notation) or even plain text.

Figure 5 - The traditional model vs. the AJAX approach [7].

Figure 1 illustrates the difference between the classic web application and the AJAX enabled

approach. The classic web application model generates a new HTML document each time the user

interface sends a request to the server. This method is expensive in regards to time and bandwidth

both to the client and the service provider as a whole page needs to be generated for the smallest of

changes to the client side [8].

The AJAX model uses JavaScript to call a new component which has been added to the model. The

AJAX engine acts as a channel between the web server and the JavaScript running on the client side.

The JavaScript is unable to call a web server directly so the AJAX engine is used to send a HTTP

request to the server which generates response based on the users request. The response is often in the

XML format which can then directly be interpreted by the JavaScript running on the client side. The

JavaScript is able to manipulate the page dynamically without the need for the page being refreshed

creating an effect that response has been generated by the users machine.


19/87


While this interactivity is being carried out in the background the user still has access to the parts of

the page which are not being updated. This method often results in faster response times and less

bandwidth being utilised as all presentation details have been excluded from the response.

There are main advantages to using the AJAX approach, but every new technology also comes with itsassociated disadvantages, AJAX is also prone to this syndrome. The major disadvantage in using

AJAX is compatibility issues between the ways certain web browsers handle the AJAX libraries. In

internet explorer 6 and below the XMLHttpRequest object is not recognised as a valid JavaScript

function instead the ActiveXObject object has to be used in order intuitively call the web server [8].

This leaves the programmer with many cumbersome validation checks to carry out before the AJAX

functions are actually used. Fortunately Internet explorer 7 now includes the support for the

XMLHttpRequest as AJAX becomes more of standardised protocol but the implications still exist for

users using older browsers.

One of the other main concerns of AJAX is network latency between calls to the server. Without

proper consideration for the size of the data being sent it could potentially leave users waiting longer

than usual for the interface to be displayed due to bottlenecks and other network related issues. This

may be confusing for some users and in turn they may end the session before the data is actually

loaded. Another implication is the when data is loaded between server calls the page layout may be

adjusted while the user is still interacting with the data.

If AJAX is used in a thought out manner it can be used as powerful tool to create a fully fledged

application over the WWW. I see AJAX as a fundamental part of Web 2.0 and as its popularity

increases it will become a day to day experience for all Web users.

2.3.3 Web feeds

Web feeds are a data format to allow users to view regularly updated information in an effective and

simplistic manner [9]. Keeping in context with the web 2.0 paradigm web feeds allow users and

developers to utilise data from more than one information source to create a unique data resource.

Typical applications of web feeds are to provide up to date information of a service such news or

weather.

These feeds are often in the XML which in such means that they were never intended to be readable in

their raw format. This gives developers a great degree of flexibility as most modern programming

languages have XML readers/parsers which can be explicitly used to manipulate and interoperate the

feed. Each feed consists of one or more elements; an element is an object of interest which contains

such information as a title, description and a link to the information provider [9]. The two main web

feed formats are RSS and Atom which are discussed later in this section.


20/87


21/87


reuse and extend them by using the web as a platform. The result of reusing and extending such

resources is known as the mashup concept [21].

Figure 5 shows the typical inputs and platforms which are associated with mashup applications. The

figure demonstrates how mashup applications are not only intended for developers it also involves thenormal daily users to interact and create their own mashups using such platforms as Facebook [22]

which allows users to create their own content and widgets [23]. It also emphasizes the division of

skill-level between the developers and normal users and how developers can reuse existing resources

to create new and interesting applications.

Figure 7 Mashup programming inputs and outputs [12].

The three key benefits of the mashup applications can be seen as;

Effective leverage of Web parts As mentioned Mashups are built on existing services,

adding code when it can't be sourced from internal or external suppliers or to provide

integration "glue" between the parts [12].

Simple, lightweight software models and services Mashups are typically built using bits of

code sourced from other vendors which usually develop all major libraries required and carry

out the computational logic. Typical examples include Google Maps or a YouTube video

player, such services originally required a massive investment from its creator [12].


22/87


A focus on self-service and DIY Mashup development is not just geared towards developers

it enables normal web users to create their own applications that could never have been

justified on a build-vs.-buy perspective. These apps can now just be created by users and

groups of collaborating users on the fly as they need them [12].

2.3.6 RSS and GeoRSS

RSS (Really Simple Syndication) is a type of web feed used to display regularly updated information

such as news or weather updates. RSS utilises the XML syntax but it only uses a predefined number of

tags in order to record the data for each element within the feed. These tags can differ depending on

the specification of RSS used. The current standard is RSS 2.0 [13]; its Predecessors include Rich Site

Summary (RSS 0.91, RSS 1.0) and RDF Site Summary (RSS 0.9 and 1.0). Figure 8 shows an example

of a minimalistic RSS 2.0 feed.

Tag Name Description RSS 2.0 ExampleTitle The title of the item.

Example Channelhttp://example.com/My example

News for September the Secondhttp://example.com/2002/09/01other things

News for September the Firsthttp://example.com/2002/09/02

Link The URL of the item.Description The item synopsis.There are also a number of optional tags which canalso be used within the item tag these include; Author Category Comments Enclosure Guid pubDate source Image

Figure 8 RSS 2.0 specification [13].

GeoRSS [14] is essentially the same thing as RSS 2.0 except it contains an extra tag which defines the

geospatial location which is relevant to each item within the feed. It also has a few other predefined

tags which are not part of the RSS 2.0 specification. This information keeps in context with the Web

2.0, information with meaning. GeoRSS is of particular importance to estateMaps because the

information for the planning alerts is actually received in this format. The planning alerts element of

the system is discussed in section 2.3.10. Figure 9 shows the structure of a GeoRSS document, the

example is actually taken from a document which estateMaps is able to process.


23/87


Tag Name Description GeoRSS Examplegeorss:point Contains the latitude and longitude

for the item in question. Thelatitude and longitude a separatedby a empty character.

....

98 Brantingham Road M16 8LZTue, 20 Nov 2007085029 ...53.44 2.25...........

...

georss:featurename Used to represent the name of theitem.

Figure 9 GeoRSS specification [14].

2.3.7 XSLT

XSLT (Extensible Stylesheet Language Transformations) is also another technology which utilises the

XML syntax. It enables one XML document to be transformed into another for example XML to RSS

or XML to HTML. XSLT can be considered to be a template processor. A template processor (also

known as a template engine or a template parser) is software or a software component that is designed

to combine one or more templates [16] to produce a single resulting document. The XSLT parser

reads the structure of the XML document to manipulate the document as specified in the XSLT file.

XSLT processing is a W3C [15] standard. All major web browsers incorporate a XSLT engine along

with all major programming languages such as Java and Microsoft C#. Figure 10 shows a high level

diagram of how template processing is actually carried out.

Figure 10 Template processor [16].

EstateMaps also incorporates a XSLT engine which is used to transform all XML documents from the

third party services into the estateMaps XML format. The design decision is discussed more during

the third section of this report. Figure 11 shows a snippet from a XSLT document which estateMaps

uses to transform GeoRSS to XML.


24/87


Figure 11 XSL code to convert GeoRSS to XML.

2.3.7 Google Maps

Google Maps [17] is a free web service which enables developers to integrate a fully interactive world

map on their web page. This map can be manipulated and navigated by the user in real time. A Google

Map has a number of advanced features some of which include;

Search for businesses and attractions.

Generate driving directions between any pair of locations.

Integrate your own data on the map.

Display information using interactive markers.

Full Ajax support.

One of the most interesting features of Google Maps is that it is able to reverse geocode for a given

address or postcode. Reverse geocoding is the process whereby an address can be mapped to its

geospatial equivalent. Unfortunately this service is rather temperamental and does not always give an

exact location for the given address hence estateMaps has its own geocoded postal information which

is discussed in section 3.4.2.

Google Maps is an integral component of estateMaps as a majority of the processing and interactions

by the user is carried out on the map. Section 3 demonstrates how the Google Maps service is actuallyintegrated into estateMaps.

2.3.8 Nestoria

As mentioned during section 2.1 Nestoria [11] is the data driving force for estateMaps. Nestoria is a

free web service which is able to query over thirty estate agents across the UK and produce an

automated reply in the XML format. Nestoria utilises the REST architecture in order for the public to

access their data and services which is explained during section 2.2.1.


25/87


Nestoria is in no way linked to selling property, they are purely a search engine and information

provider, in essence Nestoria is also a mashup application. Nestoria extracts their data from various

sources such as fish4homes [24] and pass on this information to the public. The information which

Nestoria pass on is limited for example the description of the property only contains the first two

hundred characters, but the most important data such has number of bed rooms and price are all

directly accessible. Nestoria limit the data in order to generate revenue from people accessing the

external links which are also accessible in the XML feed.

2.3.9 Flickr

Just like Google Maps and Nestoria, Flickr [18] is a freely accessible web API which enables the

public to access a library of pictures which are uploaded by its users. Flickr is considered to be one of

the earliest web 2.0 applications enabling users to share and distribute their pictures across the

internet. Unlike Nestoria, Flickr supports both the REST and SOAP architecture.

An interesting feature of Flickr and integral part of estateMaps is that flicker supports geocoding of

their images which means images can be searched by location. Previous search techniques involved

looking at the tags associated with the image which could be skewed by misinterpretations of the data.

Flickr is used in estateMaps to allow users to see scenic images which are relevant to their search

criteria by querying the entire Flickr library of images.

2.3.10 PlanningAlertsPlanningAlerts [19] is a web based API which is able to collate planning information from a number

of UK local authority planning websites. When a house developer or owner wishes to make drastic

improvements to their house they have to get approval from their local authority. Nearby residents are

able to view these planning requests and make appeals if they feel that request will affect them in any

way.

EstateMaps is able to use the information from PlanningAlerts and plot the information on a Google

Map. Users are able to see planning requests which are relevant to their search request which may

prove useful to see what type of improvements are being accepted in that area.

The information from PlanningAlerts is provided in the GeoRSS format, this means that each element

within the feed contains geospatial information as described in 2.2.6. PlanningAlerts uses the REST

architecture as described in 2.2.1.

PlanningAlerts receives its data by a process called Scraping. Scraping is the process whereby an

application reads the structure of a document i.e. HTML and extracts any relevant information.

PlanningAlerts has number of these scrapers which read the HTML of every local authority in the UK


26/87


and extract the data on a periodic basis. PlanningAlerts is then able to dynamically generate GeoRSS

feeds based on the users request.


27/87


3. DesignThis chapter details the design process of estateMaps. The first section defines the functional and non-

functional requirement which were defined for the system. The second section explains the design

methodology which was used to create estateMaps and the high level system architecture. This chapter

also details how the various third parties services are incorporated into estateMaps.

3.1 Requirements

In order to achieve the project objectives a list of functional and non-functional requirements were

drawn up. The requirements were formed from the preliminary project objectives and from the initial

research phase of the project. These requirements underline the fundamental operations which the end

application must fulfil.

3.1.1 Functional requirements

Functional requirements depict the internal operations and the behaviour of the application; they are

not concerned with the design or any performance related issues of the application.

FREQ1. The application must be able to utilise the Nestoria API in order to display properties

for sale geographically on a world map.

FREQ2. The application should allow users to search for propertys based on multiple searchconditions.

FREQ3. The house description, price and picture should be displayed where the information is

available.

FREQ4. Crime statistics about the area in which the property lies within must be displayed to

the user.

FREQ5. The application should allow multiple houses to be displayed on the map where

appropriate.

FREQ6. Caching techniques will be implemented into the application in order to reduce the

number of responses which will be generated by the server.

FREQ7. The application must be able to generate an informative response if the client does

not have all the required plug-ins into order to view the website.

FREQ8. The application should be able to display average house prices.


28/87


3.1.2 Non-functional Requirements

Non-functional requirements express the constraints upon the implementation of the application such

as performance and security related issues.

NFREQ1. The application should follow good HCI principles which will make it easier for theuser to navigate the website.

NFREQ2. The application must be able to deal with the potential unreliability of third party

services.

NFREQ3. The server should be able to generate responses concurrently based on multiple client

sessions.

NFREQ4. The application should be built in a modular design to allow for extensibility in thefuture and to cater for unreliability of third party services.

NFREQ5. The performance of the application is dependent upon a number of factors which

include the bandwidth of the user, time to generate the response, speed of the third

party services and the bandwidth available to site is host.

Many psychology studies have concluded that eight seconds, plus or minus two is the

longest which a user a user will wait before they end the session [25]. Based on this

study the application must generate a response within the ten second barrier.

NFREQ6. The application must be able to cater for cross browser compatibility in order to reach

all potential users.

3.2 Design Methodology

In order to create estateMaps the iterative/incremental design cycle was chosen. The iterative design

cycle is best suited for creating fast prototype applications or better known as beta applications. The

iterative design cycle consists of one more iterations/cycles. Each iteration consists of five phaseswhich can be modified in order to meet the requirements of the application.

Requirements This process was used to gather the functional and non functional

requirements for estateMaps. The requirements can be found in section 3.1.

Analysis & Design The analysis and design phase was combined into one phase for

estateMaps. In this section the system architecture was agreed and mock-up interfaces were

created to see how the finished application would look. The mock up interfaces can be found

in Appendix F.


29/87


Implementation Coding was initiated for estateMaps, the coding is discussed in chapter 4

Testing In this phase the application was put through black box testing. The application was

tested against a number of predefined tests. The testing is discussed in more detail in chapter

5.

Evaluation The program was evaluated and success of the application was measured against

the initial requirements.

It was decided during the preliminary planning of the project that only two iterations were needed in

order to create estateMaps because of the time limitations of the project. The development plan can be

found in appendix A.

The advantage of using the iterative approach is that incremental updates can be made to theapplication throughout its lifespan. Another advantage of using the methodology is that feedback is

received at the end of each iteration which means the application can be reassessed against its initial

requirements. At the end of each iteration the program is deployed to its intended market. In some

cases the application is actually deployed to the end users with minimal testing so that the users

become the testers. Figure 12 shows the iterative model which was the exact process which was used

to create estateMaps.

Managing and changing requirements is one of the most expensive characteristics in modern software

engineering. Fortunately the iterative approach allows for requirements to be changed or altered after

each iteration. During the second and final iteration of estateMaps it was decided that the application

should be able to display average house prices. Appendix B shows the initial requirements where as

the requirements in section 3.1 shows the requirements which were amended during the second

iteration.

Figure 12 Iterative design approach [26].


30/87


Using feedback means that there is less risk for developers and organisations in creating applications

as all resources are not spent in one go. With the standard waterfall lifecycle every step within the

development process is done in a sequence and all effort is tied towards making one final application.

Many major originations have also adopted this beta approach such as Google with Google Maps and

Microsoft with Live search [46].

EstateMaps also used this notion of risk so the most complex components of the system were actually

made during the first iteration where as the second iteration was used for perfective programming.

EstateMaps can still be seen as a beta application and functionality can be added as requirements

change.

3.3 System Architecture

EstateMaps is effectively a web based application which utilises the basic three tier architecture. Thethree tier architecture consists of three main components which are the presentation logic, processing

logic and Data logic. Figure 13 demonstrates a high level overview of the architecture used within

estateMaps and the interactions between the major components.

Figure 13 EstateMaps architecture overview.

3.3.1 Presentation logic

A majority of the presentation logic is actually carried out locally on the users machine. The main

interface into the application is the HTML file called Index, as illustrated the diagram. Index is a static

HTML file which resides on the servlet. When the user interacts with the applications GUI (Graphic

User Interface) the Index.html file is updated, but the page is never refreshed. Components on the

page are dynamically removed or added as the user interacts with the application using JavaScript.


31/87


32/87


33/87


It was decided that a fully fledged client side application would not be suitable for estateMaps mainly

because of the security restrictions of JavaScript and possible limitations of the client machines. Even

though these requests to the third party services are relatively small in size, the time required to

process them and fetch them is dependent on the clients machine and bandwidth. As mentioned in the

requirements many studies have found that eight seconds, plus or minus two is the longest a user

would wait for a typical website to load. Keeping this in mind it was decided that the three tier

architecture would be more suitable and stable for estateMaps as most of the processing and

bandwidth activities are passed to the server.

Carrying out a majority of the processing on the client side would have resulted in slow response

times and large memory requirements because of the amount of XML processing which is required.

Using the three tier architecture enables future developments to be carried out in a controlled

environment without any restrictions from the client. Figure 15 illustrates a possible client sidearchitecture for estateMaps.

Figure 15 Possible client side architecture.

3.4 Problems

3.4.1 JavaScript security restrictions

During the initial design phase and background research of estateMaps it was identified that it was not

possible to create the system as a client side application. The design decision behind this was the

security restrictions in JavaScript. JavaScript does not allow access to external servers except for that

server where the JavaScript files are hosted, which means it was not possible to send requests to the

services utilised by estateMaps. However because Google Maps use the SOAP architecture it was still

possible to send and receive requests because of the architectural style of SOAP. Hence it was decided

that a servlet was in fact needed and would be beneficial in other factors as discussed in section 4.1.Figure 16 shows a high level overview of the problem just discussed and how it was overcome.


34/87


35/87


This data was provided as both a Microsoft Excel document and SQL statements, so it could be easily

transferred into a database. Given this data it was decided that a database was needed so that the

system could validate the post code or city inputted by the user. This data would also provide useful

for mapping grid references to the crime statistics data which discussed below.

3.4.2.2 Crime Statistics Data

The UK government publish crime statistics [30] data every year which record the number of

robberies and violent attacks in every major city. This data is provided in a Microsoft excel sheet and

is available freely to the public. The problem with the data is that it is not particularly friendly and

cannot be directly transferred into a database. The second problem with the data is that there is no

geospatial data which reflects the location of the city.

Thankfully the postcode and address data maps every city to a postcode and a latitude/longitude asdiscussed in section 3.4.2.1. This information is used to map crime statistics to each postcode and city

in the UK. The crime statistics data only provides information for cities so each postcode is actually

mapped to its nearest city

Crime_DataField Type Null

crime_ID Int(5) NoForce Text NoCity Text NoPopulation figures Int(11) No

Violence_against_person_06 Int(11) NoViolence_against_person_07 Int(11) NoSexual_offences_06 Int(11) NoSexual_offences_07 Int(11) NoRobbery_offences_06 Int(11) NoRobbery offences_07 Int(11) NoBurglar_ dwelling_06 Int(11) NoBurglary_dwelling_07 Int(11) NoTheft_of_a_motor_vehicle_06 Int(11) NoTheft_of_a_motor_vehicle_07 Int(11) NoTheft_from_a_vehicle_06 Int(11) NoTheft_from_a_vehicle_07 Int(11) No

Figure 17 EstateMaps database structure.

Post_Data

Field Type NullPostcode Text NoDistrict Text NoCity (FK) Text NoX_ref Int(11) NoY_ref Int(11) NoLatitude Double NoLongitude Double NoPostcode_ID Int(5) NoCrime_ID (FK) Int(5) No

1:1

1:N

1:1

1:N


36/87


4. ImplementationThis chapter examines the implementation details of estateMaps. The various components of the

system are described in detail along with supporting diagrams which help further illustrate system

functionality. This chapter also details some of the tools used to create estateMaps.

4.1 Programming Language - The Microsoft vs. Java syndrome

In order to the implement the web server a number of different technologies could have been used

such as PHP or ASP. It was decided due to the time restrictions of the project that a technology which

I had prior experience should be considered amongst others. The two technologies which were being

considered were Microsoft C# with ASP and Java with Apache Tomcat [31]. Java was chosen for a

number of reasons, some of which are detailed below. Due to greater development experience with Java opposed to C# it was decided that more time

could be spent adding functionality and making the code more elegant rather than learning

the C# platform.

Java supports a number of free libraries which can be used to generate graphs which have to

be paid with its Microsoft equivalent.

Java supports cross platform compatibility where as Microsoft .net only has limited support in

Linux and no support in Max OSX.

PHP was also another interesting option but due to no prior scripting experience with the technology

the learning curve may have been too great.

4.2 Development Environment

Once the programming language was decided upon a suitable development environment was needed.

After some research into the different IDEs (Integrated development environment) the freely

available Netbeans [32] was chosen due to its integration with the Apache Tomcat servlet. This allows

the program to be easily tested and deployed with information and logs about the processing being

carried out by the servlet. Using the integrated server also allows the application to be deployed not

just in a test environment but also across the internet allowing multiple users to connect. Netbeans also

has a number of interesting tools which can assist with the testing of the program:

HTTP Monitor Allows the user to see information about the HTTP requests sent and

received and other useful information such as data about the processing servlet, performance

figures and the request headers.
http://.net/http://.net/


37/87


Code Refactoring Renames all variables, class location and class name if amended by the

user in the entire program.

Database Connection Monitor allows easy testing of a database connection and can be used

to see statistical information about a request such as performance ad number of requests.

4.3 Development tools

To aid in the development of estateMaps a number of tools were used. Below is a list of all the tools

used in the system and a brief description of its usage.

Netbeans As discussed in the section 4.1.2 Netbeans is used as the programming

environment for estateMaps. Netbeans is used to code the HTML, JavaScript and the server

side Java classes. Netbeans was also used to generate the UML diagrams used to explain someelements of the system in this report. Netbeans also assisted in testing the system as discussed

in chapter 5.

Apache Tomcat Tomcat is the server side container used to send and receive requests to the

servlet using the HTTP protocol.

MySQL MySQL is a local database used to store various information such postcode,

location and crime statistics data.

SQLYog SQLYog [33] is a frontend application (GUI) for MySQL which makes it easier to

interact with MySQL client.

4.4 Server Side Implementation

Figure 18 shows a UML dependency diagram for the Java server side code, which illustrates how each

class interacts with the other classes within the program. The main decision making class and entry

point into the application is the getServerResponse class. Each of the classes and their functions are

detailed on the following pages.


38/87


39/87


getServerResponse - As mentioned during the introduction to this chapter this class is the

main entry point and decision making class in the program. Its main purpose it to decide what

response needs to be generated based on the headers sent from the HTTP request.

House This class is used to store information about each house which has been found fromthe Nestoria web Service. It contains a simple constructor which sets the various variables

within the class. There are many GET methods which can be used to retrieve a particular

piece of information for a house object.

HouseXMLParse The request which the Nestoria web service generates is in the XML

format. This class process the generated XML request from the Nestoria web service and

generates a list of House objects.

XsltEngine XsltEngine is used to communicate with the external third party services. It

takes a XSL file and URL as its inputs. The URL is the location of the resource which needs

to be fetched. The URL is streamed across the internet and the response is saved. The

response is then translated into the estateMaps XML format using the input XSL file.

ImageFinder This class is used to retrieve a locationID from the Flickr web service to

retrieve images relevant to a particular area. Flickr uses a unique ID for each city or postcode

in the UK. For example the postcode M16 corresponds to keGop2.YA5qp3lp39g and

Manchester Corresponds to cTto9E.bCZ7g_w". The XML response is also parsed by thisclass. The imageFinder functionality is discussed in more detail during this chapter.

GraphGenerator This class is used to generate a dynamic graph for the crime statistics data.

The class utilises the freely available JOpenChart Toolkit [39].

HouseXMLGeneration When a new house search request is sent to the servlet a response is

expected in the XML format. This class is used to generate a XML file from a list of house

objects which are processed by the HouseXMLParse class.

PlanLocater The PlanLocater class is used to retrieve planning Information for a particular

area. It utilises the XsltEngine to convert a XML stream into the estateMaps XML planning

format.

DatabaseLink The database link class is used to create JDBC MySQL connection to access

the estateMaps database. The class creates a connection to the database and is also able to

destroy a connection once it is no longer needed.


40/87


BulidAutoCompXML This classes primary function is to query the estateMaps database in

order to find locations which could match the input text. The class is able to generate a XML

file which contains the possible matches in order to populate the predictive search bar on the

GUI.

4.5 Server Code exploration

This section explores how the various functions of the server side code are implemented and actually

work.

4.5.1 Reading HTTP Requests

In order to determine what request needs to be generated the servlet needs to be able to read the HTTP

request String. When the client side JavaScript sends a request to the server it adds a number of

parameters to the end of the URL. The request string can contain parameters with an associated value.

EstateMaps has four key requests which the server is able to process. These requests are illustrated

below with typical parameters which may be contained in the request.

..........

reqType=getHouse&locIn=Manchester&bedNum=Any&numRes=10&saleType=buy&priceVal=AnyreqType=goWidget&uIn=Manchester&saleType=buyreqType=goGraph&vi07=5891&rob07=718&sexu07=reqType=goWidget&uIn=Manchester&saleType=buyreqType=goPlan&uIn=Manchester

..........

The server side code which reads the result string is actually directly accessible using built in methods

of the Apache Tomcat servlet. The code below illustrated how the server side code is able to read the

location of the search and the request type.

.......... //Gets the location Parameter from the HTTP call associated with everycall from the JSString location = spaceRemove(request.getParameter("uIn"));

//HTTP request typeString reqType = request.getParameter("reqType");

..........

4.5.2 Processing a search request

When a user makes a new search request to locate new homes a number of events occur. Once the

request string has been processed and the relevant parameters associated with the request have been

stored, the Nestoria REST URL is manipulated with these parameters to retrieve the XML resource.

The next step is to retrieve the data from the Nestoria web service and convert the result it to the

estateMaps XML format. This is done by creating a new instance of the XsltEngine.


41/87


.................String xslIn = " estateMaps/estate_assets/house.xsl";XsltEngine getXML = new XsltEngine(xmlIn,xslIn);StringWriter convXML = getXML.transformXslt();

.................

The XSLT engine takes a URL and XSL file as its input. The XSL file is a static file which resides on

the servlet which is used to transform the document into the estateMaps XML format. The XsltEngine

streams the URL across the internet and stores the response in a temporary character stream

(StringWriter) . This stream is then converted using the inbuilt Java XSLT transformer which is also

known as TransformerFactory.

.................public StringWriter transformXslt(){

try{

xmlSource = new StreamSource(xmlUrl.openStream());

xsltSource = new StreamSource(xsltUrl.openStream());trans = transFact.newTransformer(xsltSource);trans.transform(xmlSource, result);

}catch (TransformerException ex){

ex.printStackTrace();}catch (IOException ex){

ex.printStackTrace();}return resultString;

}.................

The reason why I decided that a XSLT engine would be useful is that even if any of the third partyservices decide to change the structure of their XML feeds it means the internal operations of the

program will not have to be changed. The only part of the application which will have to be changed

is the XSL file. This means that the program will not have to be recompiled which us usually means

that the service has to be stopped. A snippet of the house XSL file can be seen below and the full

version can be found in Appendix C.

.................

.................

Select each listing in theNestoria XML file

Set the output type

Select title tag for thecurrent item

Write these tags to the file

Read the URL

Read the XSL file(local loop back)

Transform the file

Exception handling in case of errors

generated from the transformation

Location of the XSL file

New instance of the XsltEngine

Transform the file
http://www.w3.org/1999/XSL/Transformhttp://www.w3.org/1999/XSL/Transform


42/87


The result which is generated from the transform XSLT method is then sent back to the

getServerResponse class. Now that the Nestoria XML has been transformed into the estateMaps

format it needs to be parsed so each element within XML feed can be processed. There are a number

of different implementations in Java for XML parsers some of which include SAX [34], Xerces [36]

and JDOM [35]. SAX is particularly good at processing smaller sized documents without too many

layers in the document hence SAX is used within estateMaps. There is a dedicated class for parsing

this XML feed which is called HouseXmlParse.

HouseXmlParse has one specific purpose which is to read the XML feed and add each house element

into an Array List. The SAX parser is instantiated by creating a new instance of the

DocumentBuilderFactory. The DocumentBuilderFactory Defines a factory API that enables

applications to obtain a parser that produces DOM object trees from XML documents [37]. DOM is a

W3C standard which enables such documents as XML and HTML to be dynamically accessed andupdate the content, structure and style of the document [38]. The code below illustrates how the

DocumentBuilderFactory is instantiated.

................DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();DocumentBuilder db = dbf.newDocumentBuilder();dom = db.parse(new InputSource(new StringReader(xmlIn)));

.................

Once the DocumentBuilderFactory has a DOM representation of the document you are able to

directly access tags within the document. The pseudo code for reading the estateMaps converted XMLdocument is shown below. A full listing of a typical XML document generated by the XSLT engine

can be found in Appendix D.

Pseudo Code Actual code

Read all the elements of the fileGet the nodes of the Channel tag

If (the channel element has something inside){

For each element within Channel{

Get the first element of the channel tagIf (the item is not empty){

For each element within the item{

Process the current itemCreate a house objectAdd the house to the array

}}

}}

Element docEle = dom.getDocumentElement();NodeList channel = docEle.getElementsByTagName("channel");

if(channel != null && channel.getLength() > 0){

for(int i = 0 ; i < channel.getLength();i++){

Element el = (Element)channel.item(i);

NodeList p = el.getChildNodes();if(channel != null && channel.getLength() > 0){

//Iterates the channel tag find each item.for(int j = 0 ; j < p.getLength();j++){

/*Process the item in the XML document and append* the arrayList*/ Element currentItem = (Element)p.item(j);

listOfHouse.add(createHouseObj(currentItem.getChildNodes()));}

}}

}

Figure 19 Reading a transformed house XML file.


43/87


When the house element has been found in the document it is actually passed to another method

called createHouseObj. This method is used to read that particular house element in the document and

create a House object out of the data. A snippet of the code which examines the house element and

creates the house object is shown below. Figure 20 also illustrates this process diagrammatically.

.................private House createHouseObj(NodeList nodeIn){

//Iterate the XML itemfor(int g = 0 ; g < nodeIn.getLength();g++){

Element e = (Element)nodeIn.item(g);String nodeName = e.getNodeName();

//Check the node name and set the data which is applicable.if(nodeName.equalsIgnoreCase("title")){

try{

//get the title of the housetitle = e.getTextContent();

}}

//Create a house object from the data parsed.House currHouse = new House(title, description, bed, bath, price, type, type_price, link,

image, lat, longi); //return the house object.return currHouse;

.................

Figure 20 Process graph for reading a transformed house XML file.

The house class does not just store the information from the feed it also adds the crime statistics data

relevant to that property when the house object is created. This is done by retrieving the location of

that property and querying the estateMaps database in order to find the nearest city to that location.

Once the result set is retrieved from the database this information is also stored alongside the

information from the feed. The house class has a number of getter and setter methods which makes it

Gets the Title attributes value and stores it in a

temporary variable. This is done for each attribute in the

feed, until are attributes are found.

Attributes received from the XML feed are made into a

House object


44/87


easy to retrieve information from the object. The code below shows the method in the house which

retrieves the crime statics data from the database.

................ //Method to set crime stats.public void setCrimeStats(String postcodeIn){

connectionThis = new DataBaseLink(); //creates new database connectionStatement statement = null;ResultSet rs = null;Try{

Connection newMysqlCon = connectionThis.connection;statement = newMysqlCon.createStatement(); //get the crime databased on postcoders = statement.executeQuery("SELECT *` FROM estatemaps`.post_data p,`estatemaps`.crime_data c WHERE c.crime_ID = p.crime_ID and p.postcode=\""+postcodeIn+"\"");

................

Now that the Array List has been populated with the house objects from the XML feed a new XML

file must be generated with the added crime statistics data to be sent back as the response. This is

handled by the class called HouseXmlGeneration.

................ //getServerResponse class instantiating the HouseXmlGeneration to build the new XML Feed.HouseXmlGeneration sd = new HouseXmlGeneration(HouseXmlObj.getHouses())String res = sd.bulidXML();

................

HouseXmlGeneration accepts a generic Array List with the type set as House. It iterates through each

house element in the Array and builds a XML file as string. Once the XML is generated it is sentback to the waiting client side code.

................ //Public default constructor.public HouseXmlGeneration(ArrayList listOfHouse){

listToGen = new ArrayList(); //Initiate arrayif (listOfHouse.size() > 0){

//Copy the input array to the class array.listToGen = listOfHouse;

} //Initiate the result string with the XML starting tag.

xmlRepresentation = "";}

//Method to build the string based on the estataMaps schema.public String bulidXML(){

xmlRepresentation =xmlRepresentation + "";xmlRepresentation =xmlRepresentation + "";

/*iterate through each of the house objects in the array, get the house*Data via the accessor methods and append the result string.*/

for (int i=0;i


45/87


different XSL files are used transform the XML feeds. Also there is no need for the encapsulating

class such as the House class because no additional information is added for these services.

Essentially the resource is retrieved via the XsltEngine and transformed then sent back to the client

for processing.

4.5.3 Processing a Graph Request

One of the challenges in creating estateMaps is how would the crime statistics data be displayed to the

user. One of the options was to display them as raw text but because there are so many figures it

looked confusing and unreadable because they contained both the 2006 and 2007 comparisons. It was

decided that a graph would be the best possible way to display this data.

When a graph request is sent to the server side code, the crime statistics data is actually stored in the

HTTP request string as discussed in section 4.5.1. As discussed in section 4.5.2 the crime statisticsdata are actually sent to the client side when a house search request is generated. These crime

statistics data once again are actually sent back to the server side in order to generate the graph.

The reason why this design decision was taken is because instead of re-querying the database it was

actually much faster to send the statistics back in the request String. The second reason behind this is

that a graph is only generated when the user selects to see these figures for that particular property, so

they are not generated when a house search request is processed. These graphs could have been

generated when the house search request was being processed but because of the time required to

generate each graph it often consumed too many resources on the server and took too long.

The graph request is actually handled to a similar fashion as the house search request. The get

ServerResponse first processes the HTTP request which contains the crime statistics data. This data is

then passed to a dedicated class to create graphs called GraphGenerator.

.................

//Instance to the GraphGenerator classGraphGenerator graphIns = new GraphGenerator();

/*Creates a graph and stores the filename and location to the

*"filename" String*/ String filename = graphIns.graphGen(violence_against_person06, robbery_offences06,sexual_offences06, burglary_dwelling06, theft_motor06,theft_from_motor06, violence_against_person07,robbery_offences07, sexual_offences07,burglary_dwelling07, theft_motor07, theft_from_motor07);

.................

Once the GraphGenerator class receives these figures it populates a double array which is used by the

freely JOpenChart Toolkit to generate a graph. This graph is actually generated as a JPG image.

Figure 21 illustrates the structure of the double array.

Passes the variables to the GrapgGenerator,which sends back a filename to the newly

created graph.


46/87


Figure 21 Double Array graph structure.

In order to create the graph, a new ObjectChartDataModel must be instantiated. The

ObjectChartDataModel is actually the entry point into the JOpenChart Toolkit. It takes a Double

Array, Array of columns, Array of rows and a chart title as its arguments. The graph is actually stored

in a temporary location on the servlet ready to be sent to the client. The code below illustrates how the

chart is initialised.

.................

int[][] model ={

{violence_against_person06, robbery_offences06, sexual_offences06,burglary_dwelling06,theft_motor06, theft_from_motor06

},{

violence_against_person07,robbery_offences07,sexual_offences07,burglary_dwelling07,theft_motor07,theft_from_motor07

}};

String[] columns = {"Robbery offences", "Burglary dwelling offences", "Sexual offences", "Theft from avehicle", "Theft of a motor vehicle offences", "Violence against the person offences"};

String[] rows = {"2006", "2007"};

String Graphtitle = "Crime statistics 2006 vs 2007";

ObjectChartDataModel data = new ObjectChartDataModel(model, columns, rowstry{

//Export the graph to a JPGFileOutputStream storeLocation = new FileOutputStream(("\\graph_store\\"+fiName));ChartEncoder.createJPEG((storeLocation), chartObject);

//Return the filename of the graphreturn fiName;

}.................

Now that graph is has been stored in a temporary location the graph is sent back to the waiting client

byte by byte as it is not possible to send a jpg file back directly. The file is read in chunks of 1024

bytes which is sent to the client in pieces and is regenerated by the clients browser.

.................

FileInputStream in = new FileInputStream(file);byte[] buf = new byte[1024];

int count = 0;while ((count = in.read(buf)) >= 0){

out.write(buf, 0, count);}in.close();

.................

New Double Array to store both the 2006and 2007 crime statistics.

Array used to represent the columns thegraph.

Creates the new graph model.

Exports the graph to JPG image file.

Reads the file 1024 bytes at a time and sendseach piece to the client where it is

regenerated.


47/87


4.5.4 Locating Flickr images

As mentioned during section 2.3.9 the Flickr web service is able to geocode its images. This means

that images can be searched for in a specific geospatial location. Flickr has a unique ID for each city

and suburb in the UK. In order to find this ID you have to query the Flickr web service. The

ImageFinder class is used to locate this ID.

The ImageFinder class take a location as its argument. This location can either be a city or postcode.

This location is sent to the Flicker web service in a similar fashion to that used in the XSLT engine as

discussed in section 4.5.2. The response generated from the Flickr web service is a XML file which is

parsed using the SAX java parser.

.................

//GetServerResponse classImageFinder getImages = new ImageFinder();//new instance to ImageFinder

//Find the flickr location ID relevant to the locationString locationId = getImages.getLocation(spaceRemove(location));

.................

................. //ImageFinder classpublic String getLocation(String locationIn){

try{

/*URL to the flickr REST service, string is manipulated to include* location for which images are needed.*/

String flickRest = (" http://api.flickr.com/services/rest/?method=flickr.places.find&api_key=060b55702b4417d02a7b1338fc904f17&query="+locationIn+" UK");

//Converts the String type URL to a Java URL type.URL xmlUrl = new URL(flickRest); //Opens the URL to retrieve a XML file with location information.

.................

Once the response from the Flickr web service has been processed the location ID is inserted into the

Flickr image retriever URL, which is different to the location retriever. The rest of the process is

similar to that of the house search request which is discussed in section 4.5.2.

.................String xslIn = " estateMaps/estate_assets/image.xsl";

//String for flickr REST service manipulated with locationString xmlIn =(" http://api.flickr.com/services/rest/?method=flickr.photos.search&api_key=060b55702b441

7d02a7b1338fc904f17&place_id="+locationId+"&tags=city%2C+landmark&accuracy=11&safe_search=&per_page=55"); //Convert both XMLXsltEngine getGeoRSS = new XsltEngine(xmlIn,xslIn);

.................

4.4.6 Processing the search bar

As discussed during the design phase of this report the search bar on the GUI of estateMaps is able to

produce predictive results as the user inputs characters. In order to achieve this process the response

has to be generated as quickly as possible so that there is no lag between the user inserting

web 2.0 and estate maps

Documents