Automation testing

A lot of people have asked us about automated testing. This is an important subject for us since we feel like it's the most important part of my job. We know a lot about software development, so it's clearly not the best use of my time to click on the same button looking for the same dialog box every single day. Part of smart testing is delegating those kinds of tasks away so we can spend my time on harder problems. And computers are a great place to delegate repetitive work.

That's really what automated testing is about. We try to get computers to do my job for me. One of the ways we describe our goal as a tester is to put myself out of a job - meaning that we automate our entire job. This is, of course, unachievable so we don't worry about losing our job. But it's a good vision. My short term goal is always to automate the parts of our job we find most annoying with the selfish idea of not having to do annoying stuff any more.

With people new to automated testing, that's always how we frame it. Start small, pick an easy menial task that you have to do all the time. Then figure out how to have a computer do it for you. This has a great effect on your work since after you get rid of the first one that will free up more of your time to automate more and more annoying, repetitive tasks. Now with all this time you can go an focus on testing more interesting parts of your software.

That last paragraph makes it sound like writing automated tests is easy, when in fact it's typically quite hard. There are some fundamentally hard problems in this space, and those are what we are going to talk about most here. Lots of test tools try to help out with these problems in different ways, and we highly recommend finding a tool that works for you. Unfortunately all the tools we use here are custom built internal tools (it's one of the advantages of a huge development team, we have people who's full time jobs are developing test tools for us.) So I can't really recommend specific tools. But I will talk about are the two major parts of an automated test, the major problems involved, and some of the general ways to solve those problems. Hopefully it will be valuable as a way to better understand automated testing and as a way to help choose your own test tools. As a side note, implementing automated tests for a text based or API based system is really pretty easy, we are going to focus on a full UI application - which is where the interesting issues are.

Automated test can be broken into two big pieces: Driving your program and Validating the results.

Driving Your Program

This concept is pretty basic, if you want to test that when you push the spell check button a spell check session starts you have to have some way of getting your test to push the spell check button. But execution can be much trickier.

Results Verification

So you've figured out the right way to drive your program, and you have this great test case, but after you've told your program to do stuff you need to have a way to know if it did the right thing. This is the verification step in your automation, and every automated script needs this.

You have three options. You can fake it, do it yourself, or use some kind of visual comparison tool.

Faking verification is great. We don't mean that you just make up the answers, we are talking about making some assumptions about the functionality of your program and the specific functionality this automated test is checking (once again, having a well defined scope is critical.) For example when we are writing automation for the spelling engine in Visio we wrote a test that typed some misspelled text into a shape: “teh “. This should get autocorrected to “the “. But it's hard to programmatically check if “the “ was correctly rendered to the screen. Instead, we went and asked the shape for the text inside it and just did a string compare with my expected result.

Test Automation Planning and Implementation

Automation tools can greatly reduce test time when implemented correctly. They can also become shelf-ware due to frustration and non-use. In this one-day course, we focus on test automation as a software development project to be planned, budgeted, staffed, and managed for real-world software development projects. We take you through the process of choosing the right tool for your team and application, planning and managing the project, identify common pitfalls and how to avoid them through best practices and measuring results.

This class will provide a better understanding of important topics necessary for success in automation engineering organizations. From determining the best automation strategy, to writing the best tests to automate, to managing and roles and responsibilities in automation, we’ll walk you through the process and best practices of test automation.