Friday, April 24, 2009

What is Test Effectiveness?

Test Effectiveness

Test Effectiveness or bug-finding effectiveness of the test set can be measured by dividing the number of bugs found by the test set to the total bugs found in the product.

The bug-finding effectiveness of the test set can be measured by taking the ratio of the number of bugs found by the test set to the total bugs found in the product.


Where Total bugs found = Bugs found in testing + bugs found by users

Test effectiveness is important when you are evaluating the quality of the test set.

Test effectiveness only measures the percentage of bugs that the test effort found. Some bugs will be found by both the testers and the users. These are only counted once. Effectiveness only counts found bugs and is not concerned with whether or not these bugs are ever fixed. Test effectiveness is valuable when you are evaluating the quality of the test set. [Source]

As an example suppose integration testing finds 56 faults, and the total testing process finds 70 faults. Then measure of test effectiveness says that integration testing was 80% effective.

56/80 x 100 = 80%

Some more stuff on Test Effectiveness:

1. Relationship Between Test Effectiveness and Coverage.

2. On Increasing System Test Effectiveness through a Test Case Prioritization Model Using Static Metrics and System Failure Data.

3. Can Test Automation Improve Test Effectiveness?

4. Validating and Improving Test Case Effectiveness including steps to Improve Test-Case Effectiveness etc.

5. Factors affecting test-case effectiveness.


Also See:

Defect Density
Defect Removal Efficiency
Test Coverage
Schedule Variance & Cost Variance
Effort Variance