Considering the ”application stack” required to exchange data between a static web interface and remote server, one loose way to break it down is, from the perspective of the client, is;
Each of the areas constitutes a decision point in the design of a library or implementation, different approaches being available at each level.
There are two basic approaches to get a static (no reloads) web client to talk to a server;
XmlHttpRequest is the mechanism used by ScriptServer to exchange data between Javascript, running on the client and PHP, running on the server.
The XmlHttpRequest API is currently available in recent versions of Internet Explorer (5.0+ ? : not supported IE on MAC), as an ActiveX object, and Mozilla (1.4+ ? and derivatives e.g. Firefox). This already accounts for 90%+ of browsers, based on sites publishing browser usage stats. Support in Opera should begin with v7.60+ and a workaround already exists, for Opera users with Java support (see Cross Browser XmlHttpRequest http://www.scss.com.au/family/andrew/webdesign/xmlhttprequest/). Support is also available in Safari on MAC OSX.
XmlHttpRequest provides an HTTP client which can be used from Javascript to make requests to a remote server. It supports both synchronous and asynchronous calling1 and offers an API which makes simple requests easy to implement while provided enough fine grained control to cope with some of the ”trickier” aspects of HTTP (e.g. caching, char-sets etc.). Note: the XmlHttpRequest implementations used by Mozilla and IE are aware of a users browser settings such as proxy servers; there’s no need for an application using XmlHttpRequest to worry about this.
Some relevant resources on XmlHttpRequest;
The alternative and older approach is to use HTML tags which provide a src attribute, referring to a remote document to be loaded, such as <frame />, <iframe /> or <script /> (technically <object />, <img /> and <style /> may also be usable, but haven’t seen this done). The are differing variations on this approach but the all boil down to the same idea; manipulate the tag(s) with JavaScript to invoke remote requests.
This approach has the advantage of being supportable (with significant effort in writing a cross browser implementation) on older browsers but provides only basic implementation as far as an HTTP client is concerned.
Most projects which have used this approach define a local, JavaScript callback function in the static document and have this triggered by the remote document, as it loads, allowing a response to be passed asychronously to the calling client. Although effective, this has the disadvantage that the remote server is required to ”know” about the client; the content of the response must contain Javascript which triggers the clients callback function.
The request may use either the HTTP GET or POST method, typically manipulating the src URL in the case of GET while dynamically generating a form and submitting it in the case of POST.
An attempt (currently incomplete and basically a dead end for lack of time / interest in resolving browser compatibility issues) has been made, in ScriptServer, to use tag manipulation to provide an adapted XmlHttpRequest for older browsers (pseudoxmlhttp.js in source). This implementation attempts to use two hidden iframes combined with a JavaScript generated form, the ”request” iframe using the ”response” iframe as it’s submit target. Instead of having the server generate JavaScript to trigger a callback in the client, this implementation attempts to ”poll” the response frame (via window.setInterval()) for the response.
Relevant resources on tag manipulation;
Given the mechanism, the next question is what should the client and server exchange? To be more precise, how should information exchanged by the client and server be encoded.
Breaking down the available approaches (and ignoring stuff like YAML http://www.yaml.org/ and MIME-RPC http://www.mime-rpc.com/) into loose categorization, they can be summarized as;
A number of different data serialization formats exist including SOAP, XML-RPC and WDDX, which provide rules for describing data types in XML, in a platform independent manner. Some relevant resources and implementations (not already mentioned above) include;
Whether a specialized encoding format is actually necessary is a question best summarized by SOAP vs. REST http://www.xfront.com/REST-Web-Services.html (summary of most of the arguments here http://www.advogato.org/article/464.html). You could simply provide XML or even XHTML and manipulate it with DOM in JavaScript; XMLHttpRequest comes in very handy working this way - Quick tip: XMLHttpRequest and innerHTML http://www.sitepoint.com/blog-post-view.php?id=185942. This approach has been most successfully employed by Amazon’s REST API http://www.onlamp.com/pub/a/php/2003/10/30/amazon_rest.html while a hybrid form of XHTML is used by LiveSearch http://blog.bitflux.ch/archive/livesearch_got_keyboard_access.html and inserted directly into the client user interface. The Mozilla Amazon Browser http://www.faser.net/mab/ is an XUL client that demonstrates the possibilities.
A kind of middle ground between the two is using RDF (at least in the server response) to encode data and this approach is well supported by Mozilla / XUL - see theRDF Datasources http://xulplanet.com/tutorials/xultu/datasrc.html tutorial on XULPlanet.
In general XML is fine as a choice as a platform independent data format, but introduces processing overhead, particularily in some of the more complex encodings such as SOAP plus needs appropriate client / server implementations.
ScriptServer has chosen to avoid the use of XML, given it targets PHP and JavaScript specifically, where ”native” alternatives exist offering better performance and reducing complexity (see below).
A more down to earth approach, when sending data from client to server, is to use what already works with HTTP, namely encoding the URL encoding (see RFC 1738 http://www.rfc-editor.org/rfc/rfc1738.txt and URL Encoding http://www.blooberry.com/indexdot/html/topics/urlencoding.htm). Data (up to some limit in size, depending on browser and server) can be passed in the URL using a GET request or provided in the request body of a form POST (generally no size limit with POSTs), making sure the request contains the header;
For example a URL like;
Would (likely) correspond to two variables x (= 1) and y (= 2).
While easy to implement, there are three issues here. First this is (generally) a request-only encoding - what does the server respond with? Second, clients must be careful to escape values correctly (according to RFC 1738). Finally there’s the issue of how complex data structures, such as arrays and objects, are represented; although it’s possible to represent arrays and objects using encoded URLs, it requires either self-imposed rules on the exact manner URL encoding will be used or some additional encoding mechanism for values which contain complex types.
As a catch-all for everything else, a hybrid payload may be a mix of the previous two approaches. ScriptServer uses a hybrid approach which is what the discussion will focus on here2 .
A ScriptServer request, (from a JavaScript client) combines a URL encoded HTTP POST with the byte stream data representation used by PHP’s serialize() http://www.php.net/serialize and unserialize() http://www.php.net/unserialize functions.
Each encoded key / value pair in the URL encoded POST data represents an argument to the remote method being called. The names of the keys are unimportant; the order in which the pairs apprear in the encoded URL is important, and corresponds to the order of parameters in the remote method.
Each value in the URL encoded string is serialized according to PHP’s serializated data representation. This means a value could be either a scalar of complex type, containing perhaps just a string or an array of objects3 .
For the time being, objects are encoded with the class name ”ScriptServer_Object” (for which a PHP definition is provided, server-side, by ScriptServer). Future ScriptServer implementations may allow translations of JavaScript classes to PHP classes (e.g. JavaScript Date to some PHP equivalent or JavaScript DOM to PHP DOM).
The response from the server is encoded as JavaScript itself, the data structure defined as JavaScript variables and wrapped in an anonymous function. The structure is provided as a string for use with JavaScripts eval() function. For any other platform that Javascript in a web browser, exchanging data as eval()able code would be a serious security risk but because we trust the browser to execute Javascript securely (don’t we ;)), ScriptServer can get away with it.
For example this is an array inside an array, generated as a ScriptServer response (whitespace formatted for readability);
var t1 = new Array();
var t2 = new Array();
var t3 = \’maroon\’;
t2[\’name\’] = t3;
var t4 = \’80\’;
t2[\’r\’] = t4;
var t5 = \’00\’;
t2[\’g\’] = t5;
var t6 = \’00\’;
t2[\’b\’] = t6;
t1[0] = t2;
return t1;
");
Using this response in JavaScript, having fetched it using XmlHttpRequest, might look like this (exception handling omitted for example);
// Evaluate the string
var datafunc = eval(resp);
// datafunc now contains the anonymous function
// execute the function to get the data
var data = datafunc();
The reason for the anonymous function is to provide two stages of error checking as well as allowing the response payload to be assigned to a local variable (vs. creating new variables directly from the response). The call to eval() could fail if the server simply returned garbage (a server error). Once the anonymous function has been evaluated, it itself may contain an exception, generated by the server (an application error). These two possible error conditions need handling seperately.
The following is an example exception, contained in the anonymous function (formatted again for readability);
try {
throw new Server_Error(\’Invalid call syntax\’);
} catch(e) {
function Server_Error(message){
this.name = \’Server_Error\’;
this.message = message;
};
Server_Error.prototype = new Error();
Server_Error.constructor = Server_Error;
throw new Server_Error(\’Invalid call syntax\’);
};
");
In other words ScriptServer uses two encodings. PHP byte stream format for client requests and evalable Javascript for server responses. At first glance it may seem strange to use two types of encoding but the advantage is both formats have native parsers on the receiving end. In general it’s alot easier to generate some kind of encoded data than it is to parse (consider how easy it is to generate HTML vs. parsing HTML) it so avoiding writing custom parsers significantly reduces coding effort (and bugs) and reduces performance overheads.
The remaining areas of the ”application stack” relate to how a JavaScript client ”binds” to a remote server then how data from the UI populates a request and how the corresponding response populates the UI.
Publishing some service for a remote system to use introduces considerable complexity for developers, the processes of debugging and resolving errors involving, effectively, three environments; the remote server, the intervening network and the client accessing the server. What’s more, the division of logic between client and server can make changes hard to manage; a modification to the service running on the server has to be reflected in the client.
Although debugging remains difficult, it is possible to reduce the effort required by developers by adopting approaches pioneered with interface description languages http://en.wikipedia.org/wiki/Interface_description_language which describe how the service can be located and the procedures it makes public.
Inspiration for ScriptServer was taken from WSDL http://en.wikipedia.org/wiki/WSDL which can be summarized most easily be example. Using the WhoIsGoingToBePresident http://dietrich.ganx4.com/president/ service, hand coding a PHP client using PEAR::SOAP http://pear.php.net/packages/SOAP would look something like this;
require_once ’SOAP/Client.php’;
class Predictor {
var $client;
function Predictor() {
$url = ’http://dietrich.ganx4.com/president/server.php’;
$this->client = & new SOAP_Client($url);
}
function whoisgoingtobepresident() {
$params = array();
return $this->client->call(
’whoisgoingtobepresident’, $params
);
}
}
$Predictor = & new Predictor();
$prediction = $Predictor->whoisgoingtobepresident();
echo ’Kerry: ’.$prediction->kerry.’<br>’;
echo ’Bush: ’.$prediction->bush.’<br>’;
?>
By contrast, accessing this service using it’s WSDL description looks like this;
require_once ’SOAP/Client.php’;
$WSDL = & new SOAP_WSDL(’http://dietrich.ganx4.com/president/server.php?wsdl’);
$Predictor = $WSDL->getProxy();
$prediction = $Predictor->whoisgoingtobepresident();
echo ’Kerry: ’.$prediction->kerry.’<br>’;
echo ’Bush: ’.$prediction->bush.’<br>’;
?>
Simply from a lines of code count, the second example looks like less work. The client is generated automatically from the WSDL description of the service, in the latter case.
When dealing with more complex services, where published methods may take a number of arguments and the result is itself a complex variable, hand-coding the client is error prone and wasteful.
The range of remote JavaScript scripting solutions (indentified above), which represent the state of the art, generally leave it up to the developer to manual bind the client to the server.
The exception is the Mozilla SOAP / WSDL implementation which, in theory, solves the problem but it practice leaves much to be desired. What’s more, specific to PHP, is generating WSDL from a SOAP server is not as easy as it should be, WSDL being strongly typed and requiring an additional layer of (hand coded) description in PHP to translate to it.
SOAP / WSDL are designed to be platform independent; in the case of a service like Google’s, clients have been constructed is most all popular platforms that exist, including Java, .NET, Perl, Python and PHP. By contrast, we’re talking here about a JavaScript-only client primed to access a single server (in this case written in PHP). The need for platform independence is less critical; performance can be increased and layers of abstract removed by using a platform specific solution.
The approach adopted by ScriptServer is simple. Instead of an intermediate description, such as WSDL, the server is capable of generating a JavaScript client automatically, primed to access the published services. Developers building the user interface simply need to load the generated client using a tag like <script src=”http://localhost/server.php?jsclient”/> and begin using it.
In practical terms, the current implementation allows PHP classes to be ”registered” with ScriptServer then reflection http://en.wikipedia.org/wiki/Reflection_%28computer_science%29 is used to generate a corresponding JavaScript client; an object providing methods with the same names / signatures as those on the server. The generated client primed to use the server, placing no requirement on the developer to manage the networking, data encoding etc. required to access the service.
Given a client and server able to communicate with each other, how is data passed from the client UI to the server and how is a response from the server display to the client’s user?
In practical terms, with JavaScript, the first part is typically just a matter of working with the available APIs e.g.
<!--
function showValue() {
alert( document.getElementById(’example’).value );
}
-->
</script>
<form ... >
<input id=’example’ type=’text’ ...>
<input type=’button’ onclick=’showValue()’>
</form>
With access to values defined in forms etc., these can be packaged into a request for the server.
An interesting variation of this theme is the Kupu http://kupu.oscom.org/ editor which uses a document oriented approach; the client sends documents to the server. To an extent this approach is specific to Kupu, as an editing tool, but in theory it could be applied more broadly, particurily where webdav http://en.wikipedia.org/wiki/Webdav is available, fitting with the general notions of REST.
With the exception of Kupu, it’s worth noting that the client will typically send less data than it receives.
How the client uses responses it receives from the server depends largely on the nature of the data in the response.
The LiveSearch http://blog.bitflux.ch/wiki/LiveSearch approach involves generating XHTML ”snippets”, as can be seen here http://blog.bitflux.ch/livesearch.php?q=php; not true XHTML but something which can be dropped directly into the user interface using minimal DOM manipulation. This certainly makes light work on the client side but requires, at least to an extent, some awareness on the server side of how the client will use the response. That can quickly lead to violating the notions of seperation of concerns and layering, meaning logic for the client side user interface gets mixed up with server side application logic; in other words a mess which is difficult to maintain.
Another approach is to have the server generate RDF http://en.wikipedia.org/wiki/Resource_Description_Framework. This approach is used by Mozilla / XUL - see XUL and RDF: The Implementation of the Application Object Model http://www.mozilla.org/xpfe/xulrdf.htm. When building XUL applications, this has some distinct advantages, components like the <tree /> http://xulplanet.com/references/elemref/ref_tree.html being ”RDF aware”. PHP also has an excellent RDF library in the form of RAP - RDF API for PHP http://www.wiwiss.fu-berlin.de/suhl/bizer/rdfapi/. The downsides are it’s Mozilla specific and RDF is, generally, not widely understood. Also, the general sense I get of using RDF in Mozilla with an application that’s launch directly from a web site (vs. locally installed) is it takes you into the greyer, undocumented areas of Mozilla.
ScriptServer responses are packaged as native JavaScript types. This means populating the user interface with a response will typically involve either document.write() calls (see Client-Side JavaScript templates http://www.fishwasher.com/jst/page1.html) or DOM manipulation. While this can be painful, it should suit the luddites http://en.wikipedia.org/wiki/Luddite (like me) and is well charted territory.
Server responses are not restricted to native JavaScript types; it’s possible for the response to contain JavaScript function and class definitions, allowing both data and behaviour to be contained in the response. This allows for some interesting possibilities such as a response which contains a list of data as well as function definition which expects, say, an HTMLSelectElement http://xulplanet.com/references/objref/HTMLSelectElement.html as an argument and takes care of attaching the items in the list to it, as HTMLOptionElements http://xulplanet.com/references/objref/HTMLOptionElement.html. Whether this constitutes correct seperation between client and server is a matter for debate but an example is provided with ScriptServer in ”sfsearch_server.php”.