Offline JavaScript Testing

As the creator of Oplop and maintainer of its primary implementation in HTML + JavaScript, I feel an obligation to not break anything. If I screw up then someone ends up with an incorrect account password or can’t them when they need to. Me screwing up is a bad thing.

And so to help prevent that I figured I should actually test Oplop’s web app code. Now when it comes to testing web apps there are roughly two parts to a complete testing plan: unit tests and UX/end-to-end/integration/call-it-whatever-you-want-when-you-click-buttons tests. This post is purely about the former as I am lucky enough to have a web app whose UX is simple and small enough for me to test manually (although I hope to automate even those tests some day as well).

So for these previously mythical unit tests I needed a solution which would work offline (i.e. file://localhost is where all files served from) as Oplop uses the HTML application cache and so fancy XHR loaders and such are not an option. I also wanted a solution that could be used in some constant integration (aka CI) solution so that I would never accidentally break anything. This led to some digging online to figure how to hell to make this happen. It essentially broke down into three parts: the unit testing library, a way to drive those tests, and a CI solution to make sure whenever a commit occurred the tests still passed.

There are a lot of JavaScript unit testing frameworks out there. You could use QUnit from the jQuery Foundation or go more fancy and use a project like Mocha which lets you choose your own assertion library and then provides the extra stuff to write the unit tests. In the end I went with Jasmine as it just made the most sense to my brain at first reading and didn’t try to go overboard with features.

I also tossed in jasmine-jquery for good measure. That gave me some assertion functions that help with jQuery code. It also gave support for fixtures. Now this was the one place where working from file://localhost to guarantee compatibility with the application cache was annoying. Normally jasmine-jquery uses XHR to load fixtures so that you can keep them in separate files. Unfortunately XHR doesn’t like file://localhost as it views every file as its own domain from XHR’s perspective. So I had to inline string constants to get the fixtures I wanted. Not horrible, but a little annoying.

With Jasmine + jasmine-jquery I was able to write my tests and use Jasmine’s test runner to make sure my tests all executed properly. But that only gets me an HTML page in a single browser to verify the tests worked. What about running in multiple browsers at once? Or how about from the command-line for CI usage?

This is where a full-feature test runner comes into play. JsTestDriver is the old-school test driver/runner which would execute your tests on multiple browsers simultaneously and then report back the results in the terminal. I actually once used JsTestDriver with QUnit on another project so I knew I wanted something like it, but I decided to look for something newer.

I ended up finding Karma from the AngularJS team. It’s the spiritual successor to JsTestDriver and works well. It uses node.js for installation and supports all the major browsers along with file monitoring so it automatically re-runs itself upon file change. It even launches the browsers for you and shuts them down when you quit. It also includes a Jasmine interface so that in my case setup didn’t require some other package to work with the unit test framework.

With Karma I’m able to test with Blink, Gecko, and WebKit thanks to Chrome, Firefox, and PhantomJS (+ Safari on my laptop). And all of this can be done from the command-line so that it can be used from a CI solution.

I should also mention CasperJS which is a PhantomJS/SlimerJS-specific test runner that I considered. Unfortunately that leaves out Blink and so coverage isn’t quite as wide with what’s possible with Karma.

With a unit testing framework to write tests in and a test runner to run the tests in as many browsers as possible, that just left a CI solution. Luckily I had already discovered drone.io for Oplop previously for use with its Python implementation. Some people really like Travis for CI, but it’s restricted to GitHub while drone.io works with GitHub, Bitbucket, and Google Code which I appreciate since I still have projects on Google Code.

The setup was extremely simple. You can see the setup at Oplop’s drone.io page, but it’s so short I can just paste the relevant bit here:

npm install karma
sudo start xvfb
./node_modules/.bin/karma start karma.js --single-run

Basically you just install Karma, activate the X11 screen buffer implementation, and then run your tests once. That will run in Chrome, Firefox, and PhantomJS (including automatically downloading and using the newest version of PhantomJS!) and make sure the tests pass in all browsers.

Overall I’m happy with the outcome. I’ve already used the tests to make sure I didn’t botch anything when I moved a namespace-based solution for modularizing the JavaScript code. And by refactoring my JavaScript to minimize the UI assumptions it made I can test most of the UX without having to drive the browser itself, leaving me with minimal manual testing (still plan to look into getting Selenium running so I don’t even need to do that).

The biggest pain was simply figuring all of this out. Seems everyone has their own opinion and way to do this sort of thing so there’s no authoritative solution to go off of. Plus you have to realize that a unit test framework that includes an HTML page for a test runner won’t get you simple multi-browser testing nor a way to integrate into a CI solution. I’m glad I put the time into researching all of this, but geez the world doesn’t make it straight-forward if you don’t know what you are doing.