Extreme Programming: Testing at the Speed of Thought

by Serge Beaumont

Recently there's more and more talk about a new way of doing software projects called Extreme Programming (XP). Check out www.xprogramming.com for all you wanted to know about this subject and more! I won't go into XP here because there's a perfectly good internet site for it.

One aspect of XP is doing unit tests in code and running them often. This is what this article is really about: how to use a testing framework. I've already made a Delphi framework which can be found on the www.xprogramming.com site (with versions for other languages: Java, C++, VB etc.). It needs Delphi 5, but it could be adapted for earlier versions. I made this framework according to the ideas of Kent Beck, one of the founders of XP. You can find a link for the paper on the same page where the framework is located.

What is a testing framework?

Within a coded testing framework you have the advantage of having the power of your development environment at your disposal for automated testing. I don't know if you have ever used one of those "generate clicks and capture screenshots" type of tool, but they have a major failing: just one pixel changed in the GUI will make your test fail. Another thing is that testing in code gives you the possibility of acting on the error to analyse it. For instance, you could have the testing code query and parse a database table to check for values that are wrong. I've never needed such a mechanism yet, but it's nice to know that there's no limit on your testing tool.

Note that you can make two kinds of tests: tests that tests each code "module", and tests that test the functionality of the program. By manipulating a form (or a component, an MTS object, a CORBA object) in a test you can simulate user input and check to see "how far you get", i.e. how much functionality is implemented. Note that you could have a dedicated tester (QA person, whatever you call them) write these tests. This gives him or her the possibility to concentrate on designing tests, the "boring" execution of them is left to the framework. And who knows, maybe they actually like programming!

What will such a framework do for me?

The framework will accumulate a body of tests as you go along. Note that you could also start in the middle of a project if you didn't do so yet. Your testing coverage will not be complete, but whatever you have is a bonus. It should be a rule that any module (unit, class, package etc.) should be accompanied by testing classes that prove that it works according to specifications.

When the module and its test classes have been integrated into the complete project, the testing framework will help you and protect you. Whenever you make a (major) change, you can run the framework and see if anything "fell over" because of the hacking you've done. This will allow the developers to go faster without fear of breaking stuff. If you did, the framework will find out. Of course this means that you should invest in writing a complete set of tests. Ask your local QA operative, they know all about this stuff.

Another advantage is that you have hard numbers to show your customers how far you are. The numbers of tests that succeeded and failed can be shown with a percentile score. For unit tests this means that 100% should always be reached, otherwise your software doesn't work correctly. For functional tests the percentile score will gradually grow as the project goes along. More and more of these tests will succeed because the functionality was implemented.

The Testing Framework: principles

Fixture

A fixture is a fixed starting point for a test, for instance an empty TList or a TList with two TObjects in it. All testcases that belong together will start from the same (set of) fixtures. Each fixture will be an instance variable on a testcase class.

TestCase

A TestCase is defined as a group of tests that have the same fixtures. Each test is a method on a TestCase class.

Every time a test is run, all fixtures are set up before (created) and torn down (destroyed) afterwards. This is to make sure that no single test will influence another. If you want one situation to influence another, you should put it into a single test.

Note that a fixture could also be a specific database setup that could be torn down by rolling back or doing a transaction in reverse.

TestSuite

A TestSuite is used to run a bunch of tests. TestSuites can be composite, so an item in a TestSuite can be other testsuites. This enables you to build big tests from smaller ones (regression testing!).

Conclusion

This way of testing has already proven to be a major way forward (see the XP sites). I suggest that you download one of the frameworks. There's one for every major language and some lesser ones. My Delphi framework has a readme that explains the use, templates and some examples. If you use it, please let me know. I think that the framework I created is just the beginning. We can make it into something beautiful!