cupum 2007 – foz do iguaçu, brazil new information technologies for community development:...

CUPUM 2007 – Foz do Iguaçu, Brazil

NEW INFORMATION TECHNOLOGIES FOR COMMUNITY DEVELOPMENT:

Intelligent Middleware for Understanding Neighborhood Markets

Prof. Joseph Ferreira, Jr. and Dr. Raj Singh

MIT, Urban Information Systems

Department of Urban Studies & Planning

July 12, 2007

CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 2

NEW INFORMATION TECHNOLOGIES FOR COMMUNITY DEVELOPMENT

Applaud recent interest in communtiy data centers, mashups, and PPGIS But, current practice cannot meet high expectations

Problems Data maintenance and currency Limited and inflexible modeling & analysis

Solution – more IT ??!! Modular service chaining - (geo)web services, SOA Intelligent middleware for modeling and analysis Lessons (even without adding new IT)


Trends in IT for Community Development

Neighborhood Information Systems Local Data Centers serving NGOs & sharing data PPGIS (public participation GIS - beyond census tables) Public/private/non-profit partnerships Examples: NNIP website: http://www.urban.org/nnip

Mashups and Web 2.0 Internet GIS: ArcIMS, MapServer, GeoServer… Map ‘mashups’ on Google Earth/Maps, Virtual Earth,

Yahoo maps… User-generated content: Flickr photos, social networking…


GIS Office for State of MassachusettsMassGIS: http://www.mass.gov/mgis/


MassGIS:

Example of very good online

mapping and

download site for

typical US State


Google Earth Mashup: Nanaimo, Canada

City website: http://earth.nanaimo.ca/data.html NanaimoMaster and NanaimoVirtuallyDowntown using Google Earth+MapGuide


National Neighborhood Indicators Partnership: http://www.urban.org/nnip

Non-governmental organizations focused on needs of particular

communities


The Providence

Plan

Providence Rhode Island

USA


Neighborhood Knowledge Los Angeleshttp://nkla.sppsr.ucla.edu

Land Opportunities Tracking Systemhttp://lots.ucla.edu


Limitations of Neighborhood Information Systems

Keeping data up to date No problem for decennial census data Big problem for land use/ownership (cadastre)

and permitting More difficult as more parties try to share data

Interpreting & Analyzing ‘official’ data Must clean/re-interpret data for local use Hard to include ‘local knowledge’ Customized analyses are useful but non-trivial

Intelligent Middleware for Understanding Neighborhood Markets

Urban Markets Initiative Pilot Project

A collaborative effort amongMassachusetts Institute of Technology (MIT), Boston Metropolitan Area Planning Council (MAPC)Boston’s Department of Neighborhood DevelopmentThe Boston Foundation (TBF)Open Geospatial Consortium (OGC)

with support from the Urban Markets Initiative of The Brookings Institution and participation of local CDCs.

People Working on the ‘Intelligent Middleware’ Project

Principal Investigator: Joe Ferreira, Professor, MIT

Principal Collaborators: Holly St. Clair, Director, MAPC Data Center Larry Braman, Manager, Mapping and Data Services, Boston DND Charlotte Kahn, Director, Boston Community Indicators, TBF

MIT Research Team: Raj Singh, system architect (now at OGC) Jiawen Yang, database services (now Assist. Prof. Georgia Tech) Myounggu Kang, mapping services (now Assist. Prof. Seoul U.) Mi Diao, Xiongjiu Liao, Weifeng Li (PhD candidates)

Sponsored by: The Brookings Institution, Urban Markets Initiative


‘Intelligent Middleware’ ProjectGoals and Concepts

Facilitate data sharing within metropolitan areas Without everyone needing their own ‘data center’

Because everyone needs to change their copy of the data

Allow reinterpretation of official data Based on local knowledge

Without explicitly changing the original data

Codify local knowledge as rules to interpret official data on the fly

Deliver results into familiar desktop environment Tables into spreadsheet or browser; maps as mashups

Examples: Identify key landowners, targeted land uses, …


‘Smart’ Middleware AlternativeIntelligent Middleware for Understanding Neighborhood Markets

Environmental portal

Housing portal

Read-only, ‘Official’ data repositories

Intelligent data intermediaries

Normalize, trim, merge, transform, overlay, cookie-cut, extrapolate, qualify, …

Processing

Service

Service

Police Dept. Assessing Dept.

HealthDept.

HousingDept.

Data Sets Data Sets Data Sets Data Sets


One Example:Land use & Ownership Patterns

Want: REPORT & MAP of largest landowners: Top-10 Owners Keep map/report accurate and up to date Allow flexibility in definition of ‘neighborhood’ Allow flexibility in which land uses to consider (residential,

open space…) Problem: Parcel map & Assessing database not enough

Spelling problems: many different spellings of city agencies and other large landowners

Updating: old snapshot of parcels is out of date


Desired Output:

Table, Chart, and Map in Familiar Desktop Environment


Middleware Example: Prototype NGO Webpage (for DSNI)


Results BEFORE Standardizing Owner Names

For Dudley Street Neighborhood Initiative (DSNI) Neighborhood:

OWNER NAME (Boston parcels, FY2005) parcels acres

CITY OF BOSTON 182 37.85

CITY OF BOSTON BY FCL 123 9.76

DUDLEY NEIGHBORS INC 121 12.14

COTTAGE BROOK HOUSING LP 48 2.38

KEPNES EDWARD N TS 44 4.02

ALEXANDER MAGNOLIA LP 39 3.21

BOSTON HOUSING AUTHORITY 36 2.27

CITY OF BOSTON FCL 27 1.82

DUDLEY_NEIGHBORS_INC 21 1.68

RAIMONDI ROBERT A SR TS 19 0.51

Total Parcels/Acres owned by largest owners 660 75.6

…but Top-10 Landowner List has Problems:


For Dudley Street Neighborhood Initiative (DSNI) Neighborhood

OWNER NAME (Boston parcels, FY2005) parcels acres

CITY OF BOSTON 339 50.12

DUDLEY NEIGHBORS INC 145 14.06

COTTAGE BROOK HOUSING LP 48 2.38

KEPNES EDWARD N TS 44 4.02

ALEXANDER MAGNOLIA LP 39 3.21

BOSTON HOUSING AUTHORITY 36 2.27

RAIMONDI ROBERT A SR TS 19 0.51

ORCHARD PK REVITALIZATION 18 0.81

HOWARD DACIA LLC 16 1.43

ORCHARD PARK REVITALIZATION 16 2.92

Total Parcels/Acres owned by largest owners 720 81.7

Results AFTER Standardizing Some Owner Names


Solving the Spelling Problem

Problem: many different spellings of city agencies and large landowners in parcel data

Solution choices: Top-down: standardize spellings in official

datasets Bottom-up: fix spellings in copy of dataset Middle-out: accumulate spelling changes

separately and apply on-the-fly as needed…build ‘middleware’ for middle-out strategy


Example of RULE to Fix Spelling Error•Standardize OWNER names

•Create ‘update’ RULE to fix spellings•Apply on-the-fly to Boston’s parcel data (BASETABLE) to produce temporary tables (FACADES) with spelling corrections•Use temporary table for REPORTS and THEMATIC MAPS

•Example of one RULE for City of BostonUPDATE fas_parcel_bos05 SET OWNER_FY05='CITY OF BOSTON' where OWNER_FY05 like '%CITY%' and ((OWNER_FY05 like '%BOSTON%' or OWNER_FY05 like %BSOTON%') and OWNER_FY05 not like '%CITYWIDE%' and OWNER_FY05 not like '%SUITES%' and OWNER_FY05 not like '%LIGHT%‘ and OWNER_FY05 NOT LIKE '%PROPERTIES%' and OWNER_FY05 NOT LIKE '%LLC%' and OWNER_FY05 NOT LIKE '%L L C%' and OWNER_FY05 NOT LIKE '%CREDIT UNION%)';


‘Intelligent Middleware’ ProjectGoals and Concepts

Facilitate data sharing within metropolitan areas

Allow reinterpretation of official data based on local knowledge Deliver results into familiar desktop environment

HOW?

Modular design: interoperable parts, distributed access, secure access control Data intermediary manages ‘virtual’ data center middleware

Middleware uses web services and open standards

Middleware is hidden from end-user applications

User/group logins and SSL provide data sharing control


‘Smart’ Middleware AlternativeIntelligent Middleware for Understanding Neighborhood Markets

Environmental portal

Housing portal

Read-only, ‘Official’ data repositories

Intelligent data intermediaries

Normalize, trim, merge, transform, overlay, cookie-cut, extrapolate, qualify, …

Processing

Service

Service

Police Dept. Assessing Dept.

HealthDept.

HousingDept.

Data Sets Data Sets Data Sets Data Sets


Middleware Components

Concept Example DeliveryThematic MAP Map of land owned by largest

landownersGoogle Maps pinmap showing parcels owned

REPORT Table listing largest landowners in neighborhood

Excel spreadsheet showing acreage of Top-10 owners

FACADE (Virtual) parcel records with

corrected spellings Manage façade rules and sharing via browser

RULES SQL query to standardize spelling of owner names

Apply SQL rules that fix parcel data on-the-fly

Study Area Service area of Dudley Street Neighborhood Initiative

Reports are for user-defined study areas

BASETABLE Parcel data from Boston’s Assessing Office

Use read-only copy of city data


Tables are not static but result from ‘web query’:http://uis.mit.edu/umi/service/report.php?Request=QueryReport&name=top_own_parcels_a_jf1&tablename=own05_mixed_after&studyarea=dsni&xsl=http://uis.mit.edu/umi/xl/report.xsl


Same URL in Browser:http://uis.mit.edu/umi/service/report.php?Request=QueryReport&name=top_own_parcels_a_jf1&tablename=own05_mixed_after&studyarea=dsni&xsl=http://uis.mit.edu/umi/xl/report.xsl


Thematic Maps are also web services:http://uis.mit.edu/umi/mapservice/umimap.php?request=GetMap&layers=top_own_parcels_map1&WIDTH=500&HEIGHT=500&SRS=EPSG:4326&TRANSPARENT=true

Map requests conform to Web Mapping Standard (WMS) of Open Geospatial Consortium (OGC)

•Many applications now support WMS•E.g., Excel, Google, ArcGIS, …


A WMS call to the middleware services yields a thematic map top-10 landowners for all of Boston

The URL:

http://uis.mit.edu/umi/mapservice/umimap.php?request=GetMap&layers=testjf0004

The same URL works from inside of ArcMap (see

next slide)


Middleware Components

Prototype on RHEL4 Linux PC running open source applications

Database server: Postgres with Postgis spatial extensions

Mapping server: MapServer (or GeoServer) Web server: Apache2 (with Tomcat) Web services programmed in PHP Additional middleware management tools in

AJAX (asymmetric javascript and XML)


Middleware Use: More than Spelling Fixes…

Understand Land Use and Ownership Patterns: Identify “top-10” landowners in neighborhood Trace effective ownership: who controls ABC Corp. Track (un)desired land use, vacant land suitable for

reuse… Study trends: gentrification impacts of condo

conversion, predatory lending practices, …

Measure urban performance: Model affordability, accessibility, … Estimate retail ‘gaps,’ development potential, …


Conclusions: (1) Key Concepts

Sharing data within a metropolitan area Avoid ‘data center’ limitations as small geographies and

large/changing datasets explode data needs Provide mechanism for accumulating, utilizing, and

sharing local knowledge about land use, ownership, and market potential (independently of official datasets)

Use web services approach, with interoperability standards, to build ‘lego blocks’ for many management and end-user applications.

Examine the skill sets and institutional relationships that can take advantage of ‘service oriented architectures’.


Conclusions:(2) Community Development Implications

As volume and detail of ‘relevant’ data grows, traditional sharing of data and tools becomes more problematic Corrolary: Google Earth is great but can’t meet expectations

Web services and ‘middleware’ offer promising avenues for deeper sharing, but Require different mix of skills and institutional arrangements Won’t be offered soon by vendors (unless we push!) Suggests new direction and short-term strategies

More complex urban models and performance indicators have limited use unless built as plug-and-play modules that provide tunable web services.


Thanks…

Joe Ferreira, [email protected]

Raj Singh, [email protected]

‘Middleware’ project website: http://uis.mit.edu

cupum 2007 – foz do iguaçu, brazil new information technologies for community development:...

Documents

soaintelligent middleware

joseph ferreira

google earth mapguidecupum07

google earth mashup

virtual earth

communtiy data centers

google earthmaps

participation of local