Immediate Storytest Feedback
Paper and Talks
Coaching Test Automation
There has been a most surprising outcome of the development of Test
Driven Development. Bringing the tests in earlier means that they are much more
effective, for several reasons.
We are used to focusing on tests that check that a system is correct after it
has been built. But when a problem is found, there can be a considerable
delay in fixing it. First the problem has to be duplicated and understood. And
then someone has to diagnose the cause of the problem, perhaps looking at code
that hasn't been touched for several weeks or months. After they understand what
went wrong, it can be corrected. Unfortunately, often the fix introduces further
problems (in 16% of cases, on average).
Part of the problem is the delay of useful feedback. If the problem was
identified within 30 minutes of the mistake being made, it would be much simpler
to find and fix. If the delay is 10 seconds, the time lost in find and fix is
Requirements documents are usually difficult to write and difficult to read.
Someone has to understand the business requirements well and express them
clearly. Programmers have to interpret the requirements, often having to make a
best-guess at what is needed.
And then testers have to test the resulting
system, based on their interpretation of the requirements and their
understanding of the business domain and what the system does.
One approach that has found to be effective is to change the view of testing,
so that it happens all of the time, rather than after the software is done.
- Tests are used as an alternative to requirements documents, by expressing
the tests in a different form: so that they serve as executable
specifications. Some of these are written before the corresponding code is
written, and so drive the development.
- Functional, performance, and load tests can all be handled in this way
- As the system evolves, we regularly test that the system continues to
meet the requirements by running the executable specifications.
- Exploratory testing, a form of manual testing that is carried out as the
software is developed, and aims to pick up problems that have been missed by
the executable specifications.
The following are examples of what's possible. Consulting and coaching will
be tailored to your needs.
1. One Day Assessment
A one day assessment of your project(s) can provide insight into your testing
issues (both known and to be discovered), on how to solve your testing problems,
and how to make a leap forward.
2. A Few Days Onsite Coaching
- A review of your current approach
- Overview of tests as executable specifications (why and how) to the whole team.
- Half to one day coaching the Product Managers, Testers, and Business and
Systems Analysts on writing executable specifications, working with project stories
- Half to one day coaching the developers on driving development from
executable specifications, driving the domain into the code
3. Multi-day Onsite Coaching
- Present overview of tests as executable specifications (why and how) and
Fit tables to the whole team.
- Coach Product Managers, Testers, and Business and Systems Analysts for
- Chose several requirements. Start with a simple example.
I'll sketch Fit tables (storytests) on the whiteboard and alter them as we
- We may start with a simple workflow, based on use cases.
Then we'll write a second storytest for a variant workflow. Then we'll focus
on the underlying business rule and express that as a calculation rule.
- After we'd tackled a few
stories, the participants will gradually do more of the development of the storytests. I
won't formally introduce the details of Fit tables, but will show by example what could be done.
- I'll also show by example how to use
FitNesse, a tool for editing and running storytests.
- Coach testers for 1-3 days on the finer details of using storytests for
doing non-functional testing.
- Coach programmers for 2-5 days. Start with two programmers on one of
the storytests. How to write fixtures. How to drive into the code. When to
start test driving with unit tests. Driving domain into the code. Resolving
differences between current code and storytests. Handling design smells.
- Ongoing coaching, one day a week for several weeks, to locate issues and
to help with those
This can take longer than initially expected because the storytests often expose problems in the way that the software
is being developed. Coaching in OO design, TDD, and domain design may be needed
as much as coaching in storytest driven development.