At typLAB, we primarily use two programming languages: Javascript and Haskell. Haskell has a compiler, strong static typing, and is pure (no state or side-effects) by default. Javascript, on the other hand, is interpreted, dynamically typed, and, when using objects, uses lots of state (since objects are an encapsulation of state). This makes testing Javascript more important: otherwise, a typo in a variable name might go undetected until someone uses our product.
To test our code, we need a couple of things: we need to write tests, we need a framework to run our tests in, and we need to automatically run this framework regularly. The number of Javascript test frameworks is large these days, and we started by reviewing a few. We finally settled on QUnit. It provides a few simple functions to write tests, and a clean and useful page to present the results.
We then integrated QUnit into our work flow. We use a simple, home grown module system to write our Javascript. It uses keywords like Module, Class and Static to easily build modular code. We defined a new annotation, Test, to define a test in a module. All these tests are automatically collected. A separate test runner module imports the modules to test and calls a single function to run all tests in a module.
This provided us with a simple way to write tests. But if this was all, people would forget to run the tests, the tests would start failing without anyone seeing it, and the effort to write the tests would be wasted. To prevent this from happening, we wanted to integrate the running of tests in our work flow. Luckily, John Resig has written TestSwarm.
Traditionally, integrating Javascript tests in a build process or as a commit hook has been difficult, since Javascript generally depends on the browser to run. Different browsers have different quirks that might make your Javascript behave differently, and your code might also depend on theDOM, which is only available in a browser. However, you don’t want your developer machine to start many different browsers on each build or commit, and your server might not even have aGUI.
TestSwarm is a solution to this problem. It provides a central server, and new tests to be run can be pushed to this server. Clients willing to run tests can also connect to it, and are given tests to run. Results are reported back to the server. This allows TestSwarm to build reports, showing test results per browser for each test run. In our case, our server pushes a new test run to the server on each commit set that is pushed. We have a few browsers open pointing to our TestSwarm instance, which run these tests. This means that testing doesn’t get in our way, but if something breaks, we can easily see which commit broke it and on which browsers.
The final piece in our testing puzzle takes its inspiration from the Haskell world. QuickCheck is a tool that performs randomized testing. You supply a function testing a certain property of your code, and it generates random inputs of increasing size. For example, if you have written a function reversing a list, you can state the property (function) that given a list (the input argument which is randomly generated), reversing it twice produces the original list.
A Javascript version of QuickCheck exists, and we had a good use case. We use so-called reactive lists, that is, lists which can be connected to keep each other updated. For example, we can create two of these lists, and connect one to the other saying it is always the reverse of the first. When elements are inserted into the first, it sends an event to the second, which processes it, and generates an appropriate insert for the reversed list. The interesting thing is that we have an easily testable static property. Regardless of the mutations on the original list, at any point in time, the reverse of its values must be the same as the values in the output list.
QuickCheck has quickly proven its worth in testing these reactive list functions. Since it generates random values of increasing size, it quickly and reliably finds all the corner cases where bugs like to hide. We’ve written a small helper function around QuickCheck to integrate it with QUnit, so that we can write code like this:
function splitTest ()
{
var l = new R.List();
var splitfun = function (e) { return e === 1; }
var s = l.split(splitfun);
quickcheck([arbListOp(arbRange(0, 5))],
function (c, op)
{
op(l);
c.assert(QUnit.equiv(A.split(splitfun, l.unR()), s.unR()));
});
}This creates two lists: an input list l, and an output list s that is the result of splitting the input list at elements equal to 1. We then call the quickcheck function, which is our helper function. It takes a list of generators as a first argument. In Haskell, these generators can be derived from the types, but here, we have to supply them. In this case, we generate operations on reactive lists. This generator for list operations takes a generator as an argument, which it uses to generate list elements to insert. In this case we generate random integers between 0 and 4 to make sure we get at least a few 1’s.
The second argument to the quickcheck function is the test function to run. It gets a QuickCheck object as its first argument, followed by the output from all the generators. In this case it gets a function representing an operation on a reactive list. We perform this operation on the input list by calling the function. We then check if the values in the input list (the function unR performs a deep conversion from reactive to normal lists) split on 1’s are equal to the values in the output list. Here, A.split is a static splitting function, and QUnit.equiv performs a structural equality check on arrays.
Thanks to the availability of these three open source tools, setting all this up only took a few days, and has provided us with a testing setup that works, but doesn’t get in our way. In the future, we will probably also start testing DOM and user interface related code; hopefully, that will work