Abstract
This document describes how to develop web applications within the ExSite framework, how to port traditional web applications to the ExSite framework, and how to build drivers to interface to unsupported database engines.
Contents
An ExSite Web Application has the usual features associated with web applications, most importantly 2-way interactivity with the web site user, in which data is taken in, and dynamic web pages generated in response. ExSite Web Applications also have some additional features made possible by the ExSite Website Operating System framework:
There are four common types of web applications:
CGI (Common Gateway Interface) applications use self-contained programs to dynamically generate web pages. These programs are spawned by the web server when a CGI request arrives, and they generate complete web pages, which are given back to the server (and then to the client). CGI programs are brought into the ExSite framework using Dynamic Content Drivers (see below).
SSI (Server-Side Include) applications embed programs into documents that are published and served like normal HTML files. PHP is a popular web programming language that is used in this way. ASP pages also work in a similar way. Because SSI instructions are published right into web pages, they are handled in ExSite as simply another form of web page content. ExSite supports .php extensions to web page files, indicating that they should be processed by PHP before being served to the end user. Since the programming elements of your pages are just another form of content, you also gain the advantages of ExSite's Content Management features, such as versioning and access control.
Client-side applications use special browser-based languages such as Javascript, Java, and Flash, to run directly on the user's computer. This is good when fast response is needed and little contact with the server is required. Since the program (or links to the program) must be embedded into the web page, client-side applications are also another form of web page content that is managed using ExSite's Content Management tools.
Embedded Server Applications are coded right into the web server engine itself. This is typically done for performance reasons, since it is faster if the web server doesn't have to farm out dynamic page generation to other processes. ExSite supports the mod_perl mode of the Apache web server, allowing it to run as an embedded server application. This gives your web application access to the internals of the webserver as it runs. To use this functionality, your application will have to be coded as a Dynamic Content Driver (see below).
A Dynamic Content Driver (or DCD) is a way of coding a web application so that it can take advantage of ExSite's website operating system framework. They are called dynamic content drivers because:
DCDs are written in Perl, the native language of ExSite. There are methods for interfacing DCDs to applications written in other languages, if your existing code is not written in Perl.
DCDs are invoked on the public side using content management tags that are inserted into web pages. A single web page can contain DCD tags for multiple web apps. Many DCDs are like "applets", and accomplish small but useful tasks. Many different DCDs may have to work together to build a full page, by dynamically creating page elements such as:
ExSite Dynamic Content Drivers follow the model of Unix device drivers, which is to say that they support three calls:
These three calls into the DCD can be invoked using tags embedded into the raw (pre-published) HTML of the page, as described later on in DCD Calling Conventions.
The read() method is used to process all input to the DCD. ExSite provides an Input class library that will manage various forms of input in such a way that they can be shared by multiple DCDs simultaneously. This is strongly recommended over reading your own input directly, since if there are multiple DCDs at work within a single web page, one can potentially steal input that belongs to another. In particular, once stdin is read, it is gone, and the other DCDs won't be able to see the input that may have been intended for them. The Input class library handles the sharing of input between different DCDs, and is documented fully in the appendices of this document.
Among the various possible inputs that a DCD's read() method could be sensitive to are:
It is up to the application developer to determine which values are of interest to the application, and to acquire and store those values in the read() method.
The read() method will accept an optional argument string. Although not often used, this feature is available to give more control over how the read() method functions. The read() method can be called from a web page, using special content management tags, and alternative read options can be passed to it from those same tags. See DCD Calling Conventions, below.
The write() method handles the actual generation of dynamic content. Like the read method, it will accept an optional argument string, which can contain special arguments or parameters controlling the details of the requested page. The write method does not generate any output directly; rather, its output is simply assembled into a string that is returned to its caller.
The argument string that write() receives, comes from the content management tag used to invoke the application's DCD. Typically, this string will be formatted like URL-encoded form data or query string so that it can be treated like a "default" query string to initialize the DCD if no other input is provided. For example:
# written in Perl
sub write() {
my ($this,$options) = @_;
# $this is the obligatory DCD class reference
# $options is the options string from the content management tag
# use the query string preferentially, but default to the options string
my $command = $ENV{QUERY_STRING} ? $ENV{QUERY_STRING} : $options;
#...
}
The ioctl() (I/O Control) method handles all non read/write operations. It behaves like a generic function wrapper, receiving a simple string argument (the "request"), and simply does its job, returning whatever value (or values) it deems appropriate. The return value is not treated as content, and will not appear in a web page if called directly from a CMS tag. Rather the caller must know what to do with the return value. If ioctl() is called from a CMS tag in a web page, the return value will either be ignored or embedded in an HTML comment for debugging purposes.
ioctl() is used especially by ExSite to communicate with the DCD to find out what features the DCD supports. In particular, ExSite will using the following requests to find out if the DCD supports a control panel that should be placed on the adminstrator's "Web Top":
ExSite provides a BaseDCD class that can serve both as a template for writing DCDs, and also as a class that simple DCDs can inherit from to save on redundant code common to many DCDs. In particular, the BaseDCD class includes a generic constructor, new(), that will automatically invoke read() (with no special parameters) when the DCD is first instantiated. It also includes a generic read() method that prefetches any QUERY_STRING input or POST input, and stores them in the DCD object.
These generic methods are useful to many basic DCDs, meaning the developer only needs to code a write() method, telling the DCD how to generate content, and optionally an ioctl() method, if the DCD does anything extra or has an adminstrator interface.
The method for inheriting from the BaseDCD class is simply to include the following lines near the top of your DCD module:
# inherit from BaseDCD class use Modules::BaseDCD; use vars qw(@ISA); @ISA = qw(Modules::BaseDCD);
Dynamically-generated web pages can be visited recursively with different results; that is you can return to the same page using a different set of inputs, and end up with a different set of content being displayed.
This is no different in principle than a CGI program generating different output with different inputs. In fact, the page generator is a CGI program (page.cgi), so this is exactly what is happening. The important difference is that page.cgi is a clearing house for all of your web applications, so all input and output go through the same program, instead of having separate CGI programs for every function on your site.
The default version of a page is viewed using the URI to the published page, if it exists, or by using the page.cgi program to dynamically render it, ie:
http://mydomain.com/cgi-bin/page.cgi?_id=42
(This tells ExSite to display page number 42 in its database.) This page can call itself recursively, adding additional data to the URI as needed. For instance:
http://mydomain.com/cgi-bin/page.cgi?_id=42&action=view&product_id=295
Using these additional inputs, the DCD embedded in the page may make completely different decisions about what output to return to the page. This process can go on indefinitely, depending on how many variables and values your DCD can respond to.
The QUERY_STRING accepted by page.cgi is URL-encoded by convention, and the data parameters and values specified in it are visible to the internals of page.cgi and to all the DCDs that are embedded in that page.
For this reason, it helps to have some conventions to help sort out what data belongs to whom. By convention, data intended for page.cgi itself uses parameter names with a preceeding underscore (such as _id in the above URI examples). Data intended for DCDs is named with no preceeding underscore.
Since any number of DCDs can be embedded into a single page, it is normal for DCDs to see input that was not intended for them. DCDs should be programmed to gracefully ignore input that makes no sense, since that input might have been targeted at another web application on the same page.
It is also conceivable that two DCDs will see input that makes sense to both of them. It may be the case that the user is only trying to communicate with one of the DCDs, but the consequence is that both will respond, possibly with unexpected results. This is called a DCD collision. To avoid this situation:
Try to use parameter names that are likely to be unique to your DCD. Parameters like id and number are so generic that they could easily be acted upon by multiple drivers. More unique names like event_id and numWidgets are less likely to be confused.
If renaming your parameters is not feasible, be careful about sharing page space with other DCDs that accept inputs.
The control panel of an application is the administrator back-end, for configuring and managing the application. Not all applications have or need a control panel; they inform ExSite of the existence of a control panel by responding to the "ControlPanel" ioctl request. If the return value is a string scalar, that string is taken by ExSite to be a URI that will bring up the control panel. If the return value is a code reference, ExSite assumes that is a directly callable routine that will return the control panel HTML in a string. For example:
sub ioctl {
my ($this,$request) = @_;
if ($request eq "ControlPanel") {
# case 1) return URI of the control panel
return "/cgi-bin/myapp.cgi";
# case 2) return code reference to the control panel
return \&my_control_panel;
}
}
sub my_control_panel {
#...
If your application's control panel is also a form of content (that is, it can be accessed from one of your site's web pages in addition to the ExSite administrator interface), then you can simply include a hook in the write() method to call the control panel method. (This only works if your control panel method is encoded in the DCD itself; if it is an external CGI program, it cannot be used as content.) Example:
# control panel accessible from both admin interface and the site itself
sub write {
my ($this,$options) = @_;
if ($options eq "ControlPanel") {
# insert the control panel into a site page
# (probably want some access control or security checks here)
return &my_control_panel;
}
else {
...
}
}
sub ioctl {
my ($this,$request) = @_;
if ($request eq "ControlPanel") {
# admin interface has built-in security checks
return \&my_control_panel;
}
}
sub my_control_panel {
#...
A quick-and-dirty method for bringing CGI programs into ExSite is to simply run the CGI program from within the DCD, capture the content, and pass it back to ExSite for further processing. Example:
# CGI wrapper DCD
sub write {
my ($this,$options) = @_
# the options string is the URI of a CGI program
$options = /(\w*.cgi)(?(.*))$/;
my $prog = $1;
$ENV{QUERY_STRING} = $2; # careful - this changes our own query string, too!
# execute the CGI program
my $output = `$prog`;
# the ouput will include headers and extraneous HTML, so strip that gak
$output =~ s/<body>(.*)<\/body>/i;
return $1;
}
If your web application consists of Perl CGI programs, then it is relatively easy to completely absorb them into ExSite. In the simple case, you can take each Perl CGI program and convert it to its own DCD. Your original CGI program can be reformatted as a subroutine in this DCD, and the DCD write method set up to call this particular subroutine as required. Your code should be modified to only output the relevant content, since ExSite takes care of the templates and wrappers on its own.
If you have many CGI programs that act in concert to provide all of the functions of your web application, it may be more sensible in the long run to combine them into a single DCD. Here is one method to do this:
convert each CGI program to a subroutine in the DCD module. If the CGI program also contains subroutines, those are also brought into the module, of course. The main subroutine for each script must have a unique name (it's easy just to name it similarly to the original CGI program), and should contain the necessary logic to generate the DCD-specific content and layout, but not additional content (such as HTML wrappers).
give each program an action name by which it can be selected.
configure the write() method of your DCD to act as a switch statement to choose between these actions.
add an action parameter of some kind to your query strings, to select from the various options to the write() method.
If you formally invoked your web application using a series of URIs such as:
http://mydomain.com/cgi-bin/task1.cgi http://mydomain.com/cgi-bin/task2.cgi http://mydomain.com/cgi-bin/task3.cgi
Those tasks/URIs are now invoked as:
http://mydomain.com/cgi-bin/page.cgi?_id=XXX&action=task1 http://mydomain.com/cgi-bin/page.cgi?_id=XXX&action=task2 http://mydomain.com/cgi-bin/page.cgi?_id=XXX&action=task3
where the web application has been embedded into page XXX. Additional parameters to the original CGI programs can still be used with the new URIs.
Conventionally in CGI programming you would hard-code your links to related CGI functions into your output. For example:
<form method="post" action="/cgi-bin/addpurchase.cgi"> ...
If your CGI functions have all been rolled into a single package (see above), then these links can be replaced with self-referential links, eg.
<form method="post"> ...
UPDATE THIS SECTION
Well-behaved DCDs share their input with their fellow DCDs, by using the ExSite::Input class. This is especially important when dealing with POST data, since once it is read from stdin, it is no longer available for other DCDs to read. Normally a form's input will be directed at a single DCD, so it may not seem like a big deal to share it. However, if you have several DCDs in a page that can potentially respond to form input, then the first one will grab the input to see if it makes sense to it. Without shared input, subsequent DCDs will not see the input stream, even if the first DCD did nothing with it.
See the example DCD in the appendices for an example of this.
DCDs are "called" using tags embedded in the raw HTML of the ExSite Content Management System. The HTML is only "raw" before the page is fully constructed. The content management tags are replaced during the page construction process. Once a page is written to disk as a .html or .php file, it has been fully processed. No more content management tags exist in it, and no more replacements will occur. (PHP and other server-side substitutions will still occur, however, allowing a whole second level of content management to take over...)
If you have a web applet or application called "App", then it is called from your raw HTML using a tag like this:
<!--&App.method(options)-->
where "method" is one of read, write, or ioctl. For instance:
<!--&Debug.write(POST)-->
This would call our Debug applet (from the appendices, below), invoking the write() method, with argument "POST". More formally, it results in the equivalent of these Perl statements being executed:
require Modules::Debug;
my $d = new Modules::Debug;
print $d->write("POST");
Note that the only "methods" you can use in these tags are the three standard methods. All others (eg. showhash(), from the Debug example) will be ignored.
It is useful to note that if no method is given in the tag, then the write() method is assumed, so the above can be written more concisely as:
<!--&Debug(POST)-->
The default output of the DCD can be invoked with no arguments, eg:
<!--&Debug()-->
Alternative read logic can be invoked in some pages but not others by calling read() as needed, for instance (using an imaginary online shopping application):
<!--&Shop.read(currency-cookie)--> <!--display customer invoice--> <!--&Shop(action=invoice)-->
(Note that the middle line here is a normal HTML comment, which will pass through ExSite unchanged.)
In principle, you can call ioctl() using CMS tags as well. Since ioctls generate no visible output, this is only useful for executing certain tasks behind the scenes. For example, to receive an email whenever a certain page is viewed, you might use an ioctl like this:
<!--&Shop.ioctl(send_to=morgan@foo.com)-->
It is easy for one web application to call another. It simply has to include the tags to make the call as part of the output generated from its own write() methods. For example, if package A wants to invoke package B, it can do something like this:
# in package A...
sub write {
#...
return "<!--&B(options)-->";
}
This feature also makes it simple to maintain dynamic content libraries, consisting of re-useable utility functions, such as clocks, calendars, and so on, which can be re-used by any other applications. For example:
<!-- display current date and time, using the ExLib DCD --> <p>Page generated on <!--&ExLib(datetime)-->.</p>
A more direct form of calling can be performed by simply using or requiring the DCD module, and then invoking its methods directly. To ensure portability and compatibility, it is best to restrict your calls to the "public" driver methods read(), write(), and ioctl(), but in fact you are not prevented from making a call to any "private" DCD method using this trick.
# direct call to another DCD
if (require Modules::OtherDCD) {
my $dcd = new Modules::OtherDCD;
$output = $dcd->write("option-string");
}
else {
$output = "error in OtherDCD";
}
Most DCDs will have many additional methods and routines; the read(), write(), and ioctl() methods are simply the ones that ExSite uses to interface to the DCD. In some cases, you may need to split your DCD into multiple Perl packages. The additional packages can also be set up as DCDs, with their own entry points (if it makes sense to do so), or they can be stand-alone packages only invoked by the single DCD.
Example: a registration or purchasing system may have a complicated series of tools for the end-user's front-end/e-commerce system and the administrator's back-end/reporting system. Although these can be placed all together into a single monolithic package, it may make sense to keep them separate, especially if the only interaction between the sub-systems is through the database. If these functions are placed together in a single package, then you will have a single write() method that can generate both end-user and administrator content, which means you will have to be more careful about security, and make more effort to keep the code easy to read. If you broke the end-user and administrator toolkits into separate DCDs, security and readability are both enhanced. In summary, a single web application does not imply a single DCD module.
In some cases it may be necessary or desirable to create libraries or modules that are not DCDs themselves. For instance, in the above example, if there is a set of functions required by both the front and back ends, it would be best to place them in a package that is loaded by both DCDs. But if this package is placed in the DCD directory, ExSite will attempt to load it up as a driver. Although ExSite tries to handle non-conforming packages gracefully, it still results in an unnecessary performance hit. In this case, supporting packages can be placed in a subdirectory of the DCD directory. ExSite will ignore them when scanning its drivers, but the drivers themselves can load the packages as needed:
cgi-bin/Modules/Reg.pm <-- user front-end DCD to registration system cgi-bin/Modules/RegAdmin.pm <-- administrator back-end DCD cgi-bin/Modules/Reg/RegLib.pm <-- shared registration system toolkit (non-DCD)
To load the RegLib.pm toolkit, the DCDs would simply include the line use Modules::Reg::RegLib.pm; at the top of the DCD modules.
Database-driven web applications can be encoded in a DCD, using all the same programming methods described in standard CGI and web database programming guides. The most important difference is that the DCD output is returned to the caller in a string, rather than printed to stdout.
ExSite has its own database interfacing tools, which have many features that may be useful to web database programmers. See the kernel programming docs for more information on these tools. The ExSite database API is only required when interacting with ExSite's default database, which is used for basic content management. This database can be extended to incorporate the tables/functions required by the local web applications, or a second ExSite-compatible database can be created and connected to separately from the content database.
If the developer does not want to work with the ExSite database tools at all, the DCD can perform its own database connections and interactions in any way that it chooses.
Web applications written in other languages will have relatively simple Perl DCDs that simply act as a call translation layer to interface to the rest of the application.
In the simplest case, if your web applet is packaged in an executable format, you can simply spawn it as a separate process. For example:
sub write {
my ($this,$options) = @_;
# myapplet can get the HTTP request data from the environment
return `myapplet.exe`;
}
Although nice and simple, this solution is not especially high-performance, since you pay for the overhead of launching a new process and loading and executing another program. (This is no big deal in a basic CGI setup, since you've already paid this price once to start up Perl, which may be substantially more costly than your binary.) This general technique is extensible through inter-process communication methods, outlined below.
For better performance, Perl has methods for interfacing directly to C-compatible binary libraries, so that it runs entirely within the same process. For more information, see the perlxs tools (do "man perlxs" or "man perlxstut" for more info).
Perl also has a variety of language interfacing tools for various lanaguages, including C++, Java, Python, and more. The Inline tools (see Cpan.org) allow you to inline the other languages directly into your ExSite drivers, which can greatly ease the job of "gluing" your driver to a backend that is native to another language.
If your applets run externally and command line switches are insufficient to completely determine the applet's output, then some more advanced IPC tricks will have to be used.
For example, some external applications expect input (eg. commands) to come through stdin. In this case, you will have to open a bidirectional pipe to the application so that you can send the input and capture its stdout. See the perlipc manpage for more on this. Example:
use FileHandle; use IPC::Open2; $pid = open2(*myapp_read, *myapp_write, "myapp.exe"); print myapp_write "commands\n"; $output = <myapp_read>;
The unidirectional pipe (eg. open MYAPP, "|myapp.exe";) will not generally work for inlining HTML content into a page, because the output will go to stdout, and will not be captured for proper insertion into the web page templates. (It may also mess up the HTTP headers of the reply, if those haven't been output yet.) However, note that secondary content such as images are fetched in separate requests that require no templating. In these cases it is safer to use unidirectional pipes that simply dump their results to stdout.
There are other tricks that can be used for passing input and output between two separated processes, such as signals, fifos (named pipes), shared memory, and plain ol' disk files.
For high performance external applets that avoid process creation overhead (which can be costly for large programs), you can consider client-server models of construction. In essence, your applet runs continuously, and ExSite simply connects to it, sends a reqeust, and receives a reply. This can be done using sockets if you wish to follow conventional client-server methods, but if full networking functionality is not necessary, you can also use shared memory or disk files for data communication, and signals to alert your server applet that a request is being made.
The concept of a DCD is best illustrated with a simple example. Here is a simple DCD that can be used to inspect the server and application environment for debugging purposes. Note that it is written in Object-Oriented Perl, and uses the ExSite::Input class library to share input with other DCDs. In addition to the three standard DCD methods, there is also a class constructor (new()), and a private method (showhash()).
package Modules::Debug;
use strict;
use ExSite::Input;
# new : called implicitly once per page to instantiate the applet,
# when the DCD is first invoked
sub new {
my $this = shift;
my $obj = {};
my $class = ref($this) || $this;
bless $obj, $class;
# call the read method (no special args) when we create the DCD
# so that we don't have to invoke read() explicitly.
$obj->read;
return $obj;
}
sub read {
my ($this,$opt) = @_; # note: $opt not used
# use Input class to grab any form input. The Input class saves
# all input so that it can be re-used by other DCDs.
my $in = new ExSite::Input;
# retrieve and parse any data passed by POST method,
# save the data in this object
$this->{post_data} = $in->post;
# retrieve and parse any data passed by GET method
# save the data in this object
$this->{get_data} = $in->get;
}
sub write {
my ($this,$options) = @_;
# allow user to specify what info they want to see in the option string
if ($options =~ /POST/) {
return $this->showhash($this->{post_data});
}
if ($options =~ /GET/) {
return $this->showhash($this->{get_data});
}
else {
# by default, show the server environment
return $this->showhash(\%ENV);
}
}
sub ioctl {
my ($this,$request) = @_;
# The module name response is not necessary, since it is the same
# as the DCD name, which is used by default.
if ($request eq "ModuleName") { return "Debug"; }
# There is no real control panel for this module, but for the sake
# of illustration, we will display the server environment as a
# "control panel". We do this simply by passing a reference to
# the write() method, since that's exactly what it does with no
# special args.
if ($request eq "ControlPanel") {
return \&write;
}
}
# a private DCD routine that cannot be called externally from the
# administrator WebTop or from content management tags.
sub showhash {
my ($this,$href) = @_;
my $out;
# displays the keys and values of a hash
while (my ($key,$val) = each %$href) {
$out .= "$key = $val<br>\n";
}
return $out;
}
1;