Reinventing XSLT in pure Javascript

In this post we will explore some boundaries of functional programming in Javascript and show how easy it is to implement a set of combinators that can express functions similar to queries in XPath and similar to transformations in XSLT. We call the result a combinator library because we implement a few primitive queries and transformations and allow combining these into bigger ones using some basic composition functions. As we will show, all functions will follow more or less the same structure.

This post is really about Javascript, which will be the target language of this library. But most of the techniques and underlying thoughts actually come from a statically typed functional programming background. While reading this post it might be interesting to continuously keep in mind the types of the functions, which makes it much easier to understand what is going on and how this framework might be extended with more interesting transformations.

For the Haskell programmer: the framework we are about to describe here is very similar to the list arrows of the Haskell XML Toolkit. Looking at the documentation of this package might gain some additional insight.

Some functions we will see are selection functions that can be used to select parts of a document, other functions can be seen as filtering functions that exclude parts of the output, some are creation functions that introduce new structure in the output. In these post we call all of these function just transformations. The resulting framework is some hybrid comparable to XPath and XSLT.

Primitive transformations

Let’s first look at the structure of the transformations performed in XPath and XSLT. Both languages can be seen as ways of describing a function that takes one input, the context node, and produces a list of outputs, a node set. Primitive examples of such transformation functions are: getChildren (/* in XPath), getParent (.. in XPath) and getID (@id in XPath). Implementing these three functions in Javascript is very easy:

function getChildren (ctx) toArray(ctx.childNodes);
function getParent   (ctx) [ctx.parentNode];
function getID       (ctx) [ctx.getAttribute("id")];

For the ease of reading most functions in this post are written down as Javascript 1.8 lambdas. This syntax, which leaves away the braces and the return statement of functions, is currently only supported by Firefox’s SpiderMonkey engine. Note that all three functions have exactly the same type, they take one context node as input and return an array of nodes (node set) as output. There is a good reason to have a consistent type signature for these primitive transformations: this allows us to easily compose multiple primitive operations into more advanced transformations.

Sequential composition

To illustrate composition, lets see how we would like to write down a transformation that selects all the grandchildren of a context node. Ideally we would like to sequentially compose two invocations of getChildren into one. With sequential composition we mean applying some transformation to the result of some earlier transformation. Something like this:

var getGrandchildren = seq(getChildren, getChildren);

But what would the seq function look like? To make it more easy to come up with the correct implementation of the seq function it might help to write down the types of all the functions involved and work from there. We first write down the Haskell style type signatures for our getChildren function, which takes one context node and produces a list of child nodes:

 getChildren :: Node -> [Node] 

Because we want the result of the expression seq(getChildren, getChildren) itself to be transformation with the same type again, the type of seq must be:

 seq :: (Node -> [Node]) -> (Node -> [Node]) -> (Node -> [Node]) 

Which means: take two transformations as input and produce one transformation as output. By only looking at the type we can already deduce the following skeleton of the seq function:

function seq (tr0, tr1) function (ctx) /* [some nodes] */ ;

Note that this function is higher-order, it takes two functions and returns a new function. Now we only have to fill in the gap, which we can do by looking at the desired semantics. In the case of the getGrandChildren function we want to apply the first getChildren transformation to the context node and then apply the second transformation (getChildren again) over all the results of the first transformation and group together the results. Translating this to code would become something like: apply tr0 to ctx, than map tr1 over all these results and concatthe output.

function seq(tr0, tr1) function (ctx) concat(tr0(ctx).map(tr1));
function concat (xs) [].concat.apply([], xs); // stand-alone concat

Now we worked out the skeleton of the function by looking at the derived type and we have come up with an implementation by looking at the desired semantics. The function is now both type correct and works as expected!

For the Haskell programmer: we just worked out Kleisli composition for the list monad. This composition internally uses the monadic bind for the list monad instance, which happens to be the concatMap function. The concatMap function has type [a] -> (a -> [b]) -> [b], which (in our context) shows exactly how we apply a transformation (a -> [b]) to the results of another transformation ([a]) and come up with the result of the composition ([b]).

To illustrate the generic behavior of our sequential composition we use it to define some other useful transformations.

getSiblings    = seq(getParent, getChildren);
getGrandParent = seq(getParent, getParent);

Alternative composition

Now we have defined sequential composition we can also try to define another form of composition which just sums up the results of two transformations working over the same context node. For example when we want to create a transformation that selects both the grand parent and the grand children of a context node, we might want to write down something like this:

getGrands = alt(getGrandParent, getGrandChildren);

We call this composition function alt because it combines two alternative transformation paths into one. The type signature of the alt function is similar to that of seq, it takes two transformations as input and is itself again a transformation. The semantics of this transformation combinator is really easy, it just applies the two input transformation to the same context node and groups the results.

function alt(tr0, tr1) function (ctx) concat([tr0(ctx), tr1(ctx)]);

For the Haskell programmer: we just worked out the <+> function of the ArrowPlus instance for the list arrow. This function is defined in terms of the append function for lists, with type [a] -> [a] -> [a]. The semantics are very similar to that of the Alternative and MonadPlus type classes. Where the seq function is the algebraic and, product, sequence, /, times, etc, the alt function is the algebraic or, sum, alternative, |, plus, etc.

Deep recursion, filtering and creating.

Now we have two basic transformation combinators: sequential and alternative composition. Combining these two functions can get us some powerful transformation combinators. E.g. we can make the deep function that applies a transformation at arbitrary depth in document.

function deep (tr) alt(tr, seq(getChildren, lazy(deep, tr)));
function lazy (f, a) function (n) f(a)(n); // postponed function application

The deep function applies a transformation and groups the results with a recursive deep invocation for all the child nodes of the current context. Because, unfortunately, JavaScript is a strict language we have to explicitly delay the recursion in order to prevent infinite loops. Before we will test the deep transformation combinator we define three other useful primitive transformations.

function self (ctx) [ctx];

This transformation self is a bit special, it is the identity transformation which outputs a singleton list containing the context node itself. This transformation can be compared to the dot (.) in XPath.

function isElem (name) function (ctx) ctx.nodeName == name ? [ctx] : [];

This function isElem is a filter transformation function which includes or excludes a context node based on the nodeName. When the node name matches it is included as a singleton list, when there is no match the empty lists will be returned.

function mkElem (name) function (tr)
 function (ctx) [createElemWithChildren(name, tr(ctx))];

The third function mkElem is neither a selection or filtering functions, it is a creation function which takes the output node set of a transformation and surrounds it with a new element. This element will be returned as a singleton node set again.

Using the set of primitives and combinators to create a transformation that produces a simple table of contents from an input document is now very easy. The following transformation shows how to select all H1 and H2 elements from the entire document. All headers will be surrounded by a LI and inserted into an unordered list UL.

var isHeader = sum(isElem("h1"), isElem("h2"));
var toc = mkElem("ul")(seq(deep(isHeader, mkElem("li")(self))));

var myToc = toc(document.body);

Now toc is a true JavaScript function that computes a table of contents from an input document.

Comparison to XPath

Here is a (far from complete) table showing a comparison between XPath queries and our transformation functions. With the very few combinators defined, we can already rebuild a powerful set of XPath queries. Some primitives shown below, like hasAttr and precedingSiblings, have not been defined in this post. Defining them is very easy, please try it for yourself and see how far you can get with other exotic XPath axes.

XPath JavaScript . self //div deep(isElem("div")) /*/p seq(getChildren, isElem("p")) (//div|//p) sum(deep(isElem("div")), deep(isElem("p"))) //p/../.. deep(seq(seq(isElem("p"), getParent), getParent)) /p[@href]/em seq(seq(isElem("p"), hasAttr("href")) , seq(getChildren, isElem("em"))) //table/preceding-sibling::h2 deep(seq(isElem("table"), seq(precedingSiblings, isElem("h2"))))

Conclusion and goal

So we have seen how easy it is to create a powerful XML query and transformation language with a bare minimum of code. The trick is to define some primitive transformation functions and only two powerful ways to compose transformations. By using functions from one context node to a list of outputs selecting, filtering and creating elements all becomes possible. Using the composition functions you can easily build more advanced and high-level transformations like the deep function that traverses an entire document tree. The comparison between XPath and the selection primitives is quite clear. Adding an element creation function shows that the query language can quite easily become a true transformation language in which we can add new structure to the output.

While building such a library yourself is fun, it might feel a bit useless at first sight. Why reinvent XPath or XSLT while most browsers have built-in support for these tools? There are two main reasons to perform transformations this way.

The first reason is that is now very easy to add extra power to the library by plugging in new JavaScript functions. Using XSLT as a programming language, which unfortunately happens a lot in practice, almost always ends up shooting yourself in the foot with large and unmanageable recursive templates that fuzz what is really going on.

The second reason is more subtle. Here at typLAB we have shown that it is possible to change the primitive transformations and the two composition functions with their functional reactive counterparts. This enables us to incrementally rebuild the output of a transformation when only small parts of the input document change. This incremental reactivity is an extremely powerful paradigm that allows for very fast live queries over semantic and structured documents.

As you can see, this framework is very simple. Try playing with it yourself!