Developing Automation

Developing automation is like developing products. You need time to establish requirements, plan the functionality in each release, deisgn the architecture and code and test the tests. If management is not committed to treating test automation as having a valuable output, then it will never suceed. You will fall back on record and playback and very quickly you will be spending more time maintaining tests than any benefit you mayhave gained from them.

Plan big, build small. When I attack a new project, I design a soup-to-nuts solution and then I build one piece at a time based on which pieces are the most important to this project. A soup to nuts solution includes test selection, resource allocation, hang and crash trapping and recovery, extensive logging, links to source control, build and defect tracking systems, etc. In one case, the test selection piece was the most important because we had millions of tests that each took 15 minutes to run. We had to be VERY selective about which tests to run each night in order to maximize coverage over the life of the development cycle. In another case, resource selection was the key, so I built that first. But I always design the entire system, so that if I ever do build the next piece, it will fit with what is already done.

And then the automation itself needs a good design. I build my automation in four layers. The bottom (GUI) layer has one class (or include file) for each window or dialog box. All UI access goes through these files. If the UI changes in a subtle way, such as moving a control or changing a control from a listbox to a radiolist, only this class changes.

The second layer is functional actions, such as open a file, replace all instances of some text with another string, etc. Sometimes the way these are implemented changes drastically, but even then, only these functions change, not the tests. Many of these functions call other functions in this layer. Also in this layer are functions that do validation via non-standard methods, such as searching log files for errors or doing direct SQL calls to a DB.

The top layer is the test scripts themselves. Over 90% of the function calls to manipulate the product under test are to the functional layer, the rest go straight to the GUI layer. The only time I go straight to the GUI layer instead is if I'm absolutely sure no other test will ever access it in this way. Otherwise it becomes a function in the 2cd layer. One exception is the GUI tests themselves which go straight to the GUI to verify that all the dialogs and windows have the same basic layout and collection of controls.

Yes, maintaining these is a big job, but it is easier than maintaining lots of scripts.

In Visual test, the functional layer is in its own include file. In QAPartner, they are often methods in the product under test's mainwin class.

One of the powers of this methodology is that if you have a good spec, you can write the top two layers BEFORE you ever see the product. When you do get the product, you create the bottom layer, effectively "gluing" your tests to the UI. And if the UI changes, you should only have to change the bottom layer.

Once again, if management is not committing resources to test automation developement, you are probably wasting their money trying to automate and you are wasting your time as far as your career. Find someone that believes in automation.

One way to potentially get out of the trap might be to assign one person to the second or third project in the pipeline, the one you'll be testing in 3 months or so. See how much automation he or she can get done by the time code freeze hits. If you do it right, then that project should go better then most and you can assign 2 people to the another project somewhere in the pipeline.

A well designed test automation architecture should work for all of your projects. So try to get that down on paper first.


Back to my software eng page

Back to Anne and Jeremy's home page