Building Enterprise Applications with Mozilla, AMP,

XML-RPC and JSON

● Jay Sheth● http://www.moztips.com● Presentation for NYPHP Group● IBM, 590 Madison Ave● New York, NY● Tuesday, May 24, 2005

Ajax, Ajax, Ajax!What is a Firefox + LAMP application?It's an Ajax-like application

Client-Side:

* XUL * JavaScript

Server Side:* PHP* MySQL

Data Transport:* XML-RPC* JSON

AJAX-like technologies are being used by companies such as Google in Google Suggest (http://www.google.com/webhp?complete=1&hl=en ) and Google Maps ( http://maps.google.com/ ).

AJAX: Asynchronous JavaScript + XMLhttp://www.adaptivepath.com/publications/essays/archives/000385.php

A Bit of Architecture

It's All About the Central Business Logic Hub

●All business logic is centralized in the form of a central PHP-driven hub

●Both administrators and end-user can access the same business hub

●Administrators use a rich web client, delivered on-demand to Firefox

●End-users use any generic HTML-based browser

Thus the same data can be accessed and modifiedby different web clients that access a single central engine.

Thus code duplication between administrator andend-user functionality is avoided.

End-user website can be displayed using PHP 4, PHP 5or even (heaven forbid) ASP.net.

DB Abstraction at the Backend Business Hub

Putting PEAR DB to use enables easy database portability

Enough Talk – Let's See an Example Central Business Logic Hub as Implemented at

www.brooklynandbeyond.com End-user site: directory of restaurants, real estate and other businesses in Brooklyn, Manhattan, Queens, NJ and CT

Administrator's View:

Example, Part II

End-user site

Recap: The Application

* Database of restaurants and other businesses in Brooklyn, Manhattan, New Jersey and Connecticut

* Two views:Rich Client for Admin (XUL + JS + PHP + MySQL)

– Status: Beta

Flat Website for end-user (HTML + CSS + PHP + MySQL) – Status: Alpha / Proof of Concept

What's so good about a rich web client ?

● Web App + Desktop Application Hybrid● Real time response reflected in user interface● Provides 'rich', instead of 'flat' experience● Leverages Mozilla platform for improved user

interface widgets (such as editable combo boxes)

● Tabbed display paradigm enables easy editing of large number of data fields

Recipe IngredientsMaking a nice cup of XUL, JavaScript, PHP and MySQL Noodles

* Start with some PEAR libraries as the base - http://pear.php.net HTTP RequestNet SocketNet URLXML-RPCDB

* Add in Jsolait JavaScript libraries for good measure - http://jsolait.net/ *Add in PHP JSON server library - http://mike.teczno.com/json.html

* Add some PHP code to make an XML-RPC server (stir well)* Add some PHP code to make a yummy index.php file* Add some (well a lot of, actually) XUL and JavaScript code to the index.php file so that it can 'talk to' the XML-RPC server

* Open the 'content' packet, and pour it into the MySQL DB.

You have one hot cup of Mozilla LAMP noodles there!

Show us some code already !A basic XML-RPC transaction

HTTP Request:

POST /admintwo/mailingdbserver.php HTTP/1.1

<?xml version="1.0"?><methodCall><methodName>system.listMethods</methodName></methodCall>

HTTP Response:

<?xml version="1.0" encoding="UTF-8"?><methodResponse><params><param><value><array><data><value><string>getAllEntriesBrief</string></value><value><string>getAllCategories</string></value><value><string>getTelLookupData</string></value></data></array></value></param></params></methodResponse>

Cool Feature: Auto-address lookupusing reverse phone lookupStep 1) Enter a phone number

Auto-address lookup (II)Step 2) Press the reverse phone lookup button

Code for Auto-address Lookup (I)

XML-RPC Server-Side Code

require 'google_lookup.php';/*Use Google to do a reverse lookup of a telephone number, to get the contact information associated with that number.Return that information.*/function get_tel_lookup_data($params){

$tel_no = XML_RPC_decode( $params->getParam(0) );

// Check the telephone number for valid syntax before performing a lookup$tel_parts = extract_tel_digits($tel_no);

if ( ! is_array($tel_parts) ){

return new XML_RPC_Response(0, $XML_RPC_erruser + 1, "Error: phone number invalid.");}

$my_lookup = lookup_google('http://www.google.com/search', $tel_no);$retval = XML_RPC_encode($my_lookup);return new XML_RPC_Response($retval);

}

Code for Auto-address Lookup (II)

Google lookup function (excerpt)require_once 'HTTP/Request.php'; // PEAR HTTP_Request Library to POST data (or to GET data)

function lookup_google($remote_server, $telephone){$req =& new HTTP_Request($remote_server);$req->setMethod(HTTP_REQUEST_METHOD_GET);$req->addQueryString('q', $telephone);

if (!PEAR::isError($req->sendRequest())) { $response2 = $req->getResponseBody(); } else { $response2 = 0; }

if (! empty($response2) ){

$num_lookups = 0;$our_matches_all = array();$response_no_tags = strip_tags($response2);$response_no_tags = str_replace(' & ', ' and ', $response_no_tags);

// Make space after Mapquest links so that after HTML tags have been stripped, there is a space// before the business name

$response_no_tags = str_replace('MapQuest</a>', ' ', $response_no_tags);

$num_matches = preg_match_all('/([A-Za-z\s-]+),\s\((\d{3})\)\s(\d{3})-(\d{4}),\s([0-9-]+ [\w\s.]+[.#0-9\s]*),\s([A-Za-z\s]+),\s([A-Za-z]{2})\s(\d{5})/', $response_no_tags, $our_matches_all, PREG_PATTERN_ORDER);

/*More stuff happens here ...*/

// Populate and return new array with relevant information from $our_matches_all }

}

Code for Auto-address Lookup (III)XUL widgets population / JavaScript code

function getTelLookupData(tel_no){

set_waitbox('-- please wait --');var methods = [];try{var server = new xmlrpc.ServerProxy(globalXmlRpcServer, methods);var lookup_data = server.getTelLookupData(tel_no); // calling remote XML_RPC function here!// Automatic type conversion here from PHP assoc array to JS objectif ( lookup_data['rpc_msg'] != 'No reverse lookup matches found.' && lookup_data['rpc_msg'] !=

'Reverse Lookup: an error has occured.' ){document.getElementById('b_name').value = lookup_data['b_name'];document.getElementById('b_phone').value = lookup_data['b_phone'];last_phone_num_lookup = lookup_data['b_phone'];document.getElementById('b_street_address').value = lookup_data['b_street_address'];document.getElementById('b_city').value = lookup_data['b_city'];document.getElementById('b_state').value = lookup_data['b_state'];document.getElementById('b_zip').value = lookup_data['b_zip'];}else{

// Turn off auto lookup if no phone number has been found, or if an error has occuredtoggleReverseLookup();toggleALButton();

}set_waitbox( lookup_data['rpc_msg'] );}catch(e){set_waitbox('Error:' . e);alert(e);}

}

Automagical zip population

Similarly, when an entry is made and both zip codes are left blank, they are looked up.

If an appropriate match could be found, this information is saved to the database.

Otherwise, an error is exposed as a JavaScript alert, and a color and text change in the notice box. (More on the notice box in a later slide.)

Error Checking

* Centralized on the server side* Exposed on the client side as native JavaScript alerts / actions

Nice feature: editable drop-downs

XUL has editable drop-downs – a big improvement over HTML's select tag

XUL Code for Editable Dropdowns

<hbox><label control="b_neighborhood" value="Neighborhood:"

style="width:100px;" />

<menulist editable="true" id="b_neighborhood"><menupopup>

<menuitem label="--select--" value="--select--"/>

</menupopup></menulist>

</hbox>

Note: neighborhood values are dynamically added using the DOM viaJavaScript

JavaScript Code for Dynamic Neighborhood Drop-down Population//Get a list of distinct neighborhoods from the entries table

function getAllNeighborhoods(){

var methods = [];try{var server = new xmlrpc.ServerProxy(globalXmlRpcServer, methods);var all_neighborhoods = server.getAllNeighborhoods(); // calling remote XML_RPC function here!

// Automatic type conversion here from PHP assoc array to JS object

var menuList = document.getElementById('b_neighborhood');var menuList2 = document.getElementById('s_near_neighborhood');

for (var i in all_neighborhoods){

aneigh = all_neighborhoods[i];neighborhood = aneigh['neighborhood'];

if (neighborhood) {

menuList.appendItem(neighborhood, neighborhood);menuList2.appendItem(neighborhood, neighborhood);

}

}

}

catch(e){alert(e);}

}

PHP Code for Dynamic Neighborhood Drop-down Population

//Get a list of all distinct neighborhoods in the entries table

function get_all_neighborhoods(){

global $username, $password, $server, $database, $mailing_table;global $XML_RPC_erruser;

$dbh = DB::connect("mysql://$username:$password@$server/$database");

if ( DB::isError($dbh) )// CHECK PEAR DB CONNECTION ERRORS{

// Return XML-RPC faultreturn new XML_RPC_Response(0, $XML_RPC_erruser + 1, "DB Error: Could not connect to server.");

}else{

// Retrieve list of distinct neighborhoods$dbh->setFetchMode(DB_FETCHMODE_ASSOC);

$query = "SELECT DISTINCT neighborhood FROM $mailing_table ORDER BY neighborhood ASC";$mlist_neigh = $dbh->getAll($query);

if ( DB::isError($mlist_neigh) )// CHECK FOR PEAR DB / QUERY RUN ERRORS{

return new XML_RPC_Response(0, $XML_RPC_erruser + 2, "DB Error: Could not run query.");}else{

$retval = XML_RPC_encode($mlist_neigh);return new XML_RPC_Response($retval);

}}

}

Nice feature: PEAR QF-like hier-select menus

Step 1: select category from editable-dropdown

Step 2: select from list of subcategories pertaining to this category

Hierselect (Part II)

Code for XUL Hierselect Menus

Code Link :http://www.moztips.com/wiki/index.pcgi?page=XuLHierMenulists

Live Demo:http://www.moztips.com/code/mozbugs/menulist/menulist_bug.php

Capitalizing on MySQL's FULLTEXT indexingfor listings search

●Deliberate compromise in DB design: de-normalized table structure ●Category, Subcategory, Country Affiliation etc. stored in same table●These fields stored as VARCHAR(255)●Created a MySQL FULLTEXT index for these, and other fields

●MySQL is great at storing, searching through and filtering data

Sample search queries:●Bronx Italian●Bay Ridge pizza●Japanese restaurants●Vegetarian●West Village Italian

Sample FULLTEXT query (simplified):SELECT id, MATCH (title,body) AGAINST ('Tutorial') FROM articles;

More info.:http://dev.mysql.com/doc/mysql/en/fulltext-search.html

MySQL FULLTEXT Search Example (I)

Content Administrator's View

MySQL FULLTEXT Search Example (II)

End User's View

Problem with rich web applications and large datasets:

things can slow down, instead of speeding up

Things slow down because:

●Large amounts of separate pieces of data are loaded on each 'event' (e.g. editing an entry)●Choosing XML-RPC as the underlying data transport is convenient but also expensive●Parsing XML on the client-side is resource-intensive

How to speed things up:

●Optimize application design for speed●Reduce the amount of requested data by using pagination●Use an alternative data transport format, such as JSON (JavaScript Object Notation)

Techniques for Optimizing Rich Web Applications for Speed

●Intelligently cache data locally in either JavaScript arrays or XML files●Local XML file caching is more complicated, as Firefox requires local file access●Ensure that JSolait library does not request list of remote methods on each web service request●When data for one listing is retrieved, also preload data into JavaScript array for surrounding 10 listings●Only reload data from remote DB when data on the client side is stale

Case in point: reloading complete listings list takes too long

(On to-do list)

Using JSON for Optimized Data Transport Speed

JSON: JavaScript Object Notation

●JSON is a JavaScript Object literal●JavaScript objects are the same as JavaScript associative arrays

Example:{"myname":"Jay","place":"New York"}

PHP Equivalent:array('myname' => 'Jay', 'place' => 'New York')

How to Get Data via JSON

* Mozilla makes a GET request to page such as: http://localhost/json/test_json.php?sort=az

* That PHP page queries MySQL and lists all entries (such as weblog posts and ids) in alphabetical order in JSON format , using the JSON PHP library

* Mozilla reads this object literal string from the PHP-generated page, converts it into a JavaScript Object / Assocative Array, using the eval() function

* Mozilla loops through the JavaScript Object, and enters each title into a XUL Listbox

Using JSON results in a 600 - 700% speed increase over XML-RPC.

Some JSON Request Code (I)

init_urllib.js

// init_urllib.jsvar urllib=null;try{ var urllib = importModule("urllib");}catch(e){ reportException(e); throw "importing of urllib module failed.";}

Some JSON Request Code (II)HTML / PHP Page:<!-- This page is:http://localhost/json/test_json.html /http://localhost/json/test_json.php--><script type="application/x-javascript" src="jsolait/init.js"> </script><script type="application/x-javascript" src="jsolait/init_urllib.js"> </script><script type="application/x-javascript" src="jsolait/lib/urllib.js"> </script>

<script type="application/x-javascript">var rslt = urllib.sendRequest("get", "http://localhost/json/test_json.php");

if (rslt.status == '200'){jsonSerialized = rslt.responseText;

//eval( 'obj3 ={"myname":"Jay","place":"New York"};' );eval( jsonSerialized );document.write(obj3['myname']);document.write(obj3['place']);}else{

alert('Error: the server could not find the specified file.');}</script>

Some JSON Request Code (III) PHP Script (loaded via GET HTTP Request)

<?php// test_json.php

require_once('JSON.php');$json = new JSON();

// convert a PHP value to JSON notation, and send it to the browser

$value = array('myname' => 'Jay', 'place' => 'New York');// $value is:// {"myname":"Jay","place":"New York"}$output = $json->encode($value);$output = 'obj3 =' . $output . ';';print($output);?>

Powering the End-User Site with PEAR XML-RPC Clientand PEAR Sigma Templating System (I)

(bonus material)

Powering the End-User Site with PEAR XML-RPC Clientand PEAR Sigma Templating System (II)

(bonus material)

<?php// Use MySQL's fulltext searching to search DB$s = $_REQUEST['s'];// Include preference file, PEAR XML-RPC Client, and PEAR Sigma Template Library // Instantiate Sigma template object. Set the templates directory to the subdirectory called 'templates'

if ( strlen($s) > 0 ){

$client = new XML_RPC_Client($xmlrpcpath, $xmlrpcserver, 80);$client->setCredentials($xmlrpcusername, $xmlrpcpassword);

$message = new XML_RPC_Message('getEntriesByKeywordgetEntriesByKeyword');$message->addParam( new XML_RPC_Value($s, 'string') );$response = $client->send($message);

$return_value = $response->value();$search_results = XML_RPC_decode($return_value);

$num_s = count($search_results);for ($j = 0; $j < $num_s; $j++){

$row = $search_results[$j];$tpl->setVariable(

array( 'name' => $name, 'id' => $row['id'], 'street_address' => $row['street_address'], 'address2' => $row['address2'], 'cross_streets' => $row['cross_streets'], 'city' => $row['city'], 'state' => $row['state'], 'neighborhood' => $row['neighborhood']) ); $tpl->parse('sitem_block');}$tpl->show(); // Display the newly created HTML file

}else {$tpl->show(); // Display the newly created HTML file}?>

PHP Fulltext Search Code Excerpt (some code omitted!)

Powering the End-User Site with PEAR XML-RPC Clientand PEAR Sigma Templating System (III)

(bonus material)

Sigma gh

Sigma Template Code Excerpt

<html><head>

<title>Brooklyn and Beyond </title></head><body><div class="mainbody">

<h3> Search Listings </h3>

<p>{num_results_msg}</p>

<p>{no_results_msg}</p>

<p> {search_tip} </p>

<!-- BEGIN sitem_block --><ul> <li>

<strong><a href="detail.php?id={id}">{name}</a></strong> <br /> {street_address} <br /> {cross_streets} <br /> {city} <br /> {state} <br /> {neighborhood} </li>

</ul><!-- END sitem_block -->

</div></body></html>

Conclusions

●AJAX-like applications can dramatically improve user experience●XUL + JavaScript on the client is perfect for data entry tasks●Having each component separated from the other makes for 'hot-swappable' components, and an easy-to-upgrade site

Hot-swapping components – some examples:If you (or the pointy-haired one) decide tomorrow that it would be way cool to use ASP.net to display the end-user site, you could easily do so. Simply acquire an XML-RPC client for ASP.net, and rebuild the end-user portion. The central business hub will still be powered by PHP 4 or PHP 5, and the ASP.net site will access data via the same XML-RPC API.

Suppose you would like to migrate from MySQL to PostgresQL for the datastore. Simply import your data into PostgresQL, change the PEAR DB DSN strings in the central business hub, and modify queries slightly to account for the differences between MySQL and PostgresQL.

It's as easy as that! No, really!

Questions? Comments?

Thank You!