Saturday, July 4, 2009

Can software test automation improve test effectiveness?

Existing metrics techniques such as code coverage can be used to estimate or compute test effectiveness before and after automation. Automated tests can be incredibly effective, giving more coverage and new visibility into the software under test. It also provides us with opportunities for testing in ways impractical or impossible for manual testing, yet conventional metrics may not show any improvements. Automated tests can generate millions of events and sequences limited only by the machine power and time available for running the tests. These tests can find defects in code already 100% covered. Employing random numbers allows sampling of events and sequences, and also allows tests to do new things every time they are run. Automated probes can look inside the product being tested at such things as intermediate results, memory contents, and internal program states to determine if the product is behaving as expected.