Rick Mugridge, 15 July 2011

When it takes ages for tests to run, delaying the build, it's tempting to look at running them in parallel.

We look at some techniques that can be used to parallelise test execution.

We'll see that awkward interactions between our tests are the major limiting factor to speeding things up. And that we'll have to choose the right tradeoff between speed and the various costs.

1. Are Your Tests Ready?

If your tests are not in a good state, it's going to be difficult to parallelise them. It may be hard enough anyway.

First, be sure that:

Rather than trying to parallelise all the tests at once, a good strategy is to do it incrementally. Pick a few of the slower-running tests, ready them, and parallelise those first. This will give you experience with the issues, making it easier to parallelise further tests (or not, if it's too hard).

2. Running on Separate Machines

The easiest way to speed up your tests is to run sub-suites on different machines, each containing a complete test environment consisting of:
Each test machine runs through its own sub-suite in sequence. If you have ten machines, you can possibly reduce the test execution time to a tenth of what it was before.

The speedup that can be achieved is usually limited by the cost of each additional test system, whether they are handled on local machines or on a cloud. And there is a cost to setting up and managing multiple machines, regardless of where they are.

To use this approach, we now have to:
Over time, we have to:
But there are some limitations to this approach, as we see in the next section.

2. Running on Separate Machines, With Shared Backends

It may not be possible to have all backend systems running on each machine, due to:
If some backend systems have to be shared, this approach can only work if:
We now have to also:

3. Multiple Processes per Machine, with Shared Backend

It's possible to have several processes running a test environment on each machine, with sharing of (some) backend systems on each machine or across machines. But the bulk of the test environment will still be duplicated in each process, so the savings are not likely to be that great.

You need to select the tests that run in each of the processes to avoid clashes on the shared systems on each machine.

This approach can only work if all of the above restrictions are met, and:
We now have to also:

4. Multiple Threads, with Shared Platform, Test Framework and Application

We can share more of the test environment by running multiple threads on each machine. This can allow us to speed things up without increasing the number of machines, as long as the tests are not compute-bound.

The number of threads we can run without introducing extra delays is determined by the number of cores available and by the inherent delays in the execution of each of the tests due to delays in backend processing and I/O.
This allows us to share much more, making a big difference to the memory footprint. But again, sharing leads to potential problems.

This approach can only work if all of the above restrictions are met, and:
Any system is not thread safe if it makes use of global variables (such as statics in Java or C#) or global resources without specifically allowing for such sharing (such as with locks).

We now have to also:

5. Summary

While it is desirable to speed up test execution, there are tradeoffs in doing so.

As we saw, the easiest way to speed up your tests by a factor of 10 is to run 10 (or so) separate machines in parallel. However, each machine needs to contain a complete copy of the test environment, including all backend systems. Scaling this up to 1000 in parallel is not always possible, due to cost or other constraints.

To reduce the costs, we can introduce sharing of resources. Unfortunately, the more we share, the harder it is to ensure that our tests run independently. When tests fail, it's more difficult to diagnose the fault when we use sharing.

Managing test suites for parallel execution takes real effort. It's best to organise tests around the structure of the application domain, so it's unfortunate to have to reorganise test suites for test execution. It's possible to use tags to specify sets of tests that cannot run at the same time, but there are no FitNesse-based tools available that do that at this point.

The easiest tests to parallelise are those that don't change the "global" state of the application system and backend systems at all. They are relatively cheap to parallelise, but there are usually few such tests.

5.1 Other Approaches

There are other approaches that can be used: