Testing in Agile
This post contains:
Traditional Style Quality Assurance
Agile approaches are changing the conversation about software development
An Agile Tester
A professional tester who embraces change, collaborates well with both technical and business people, and understands the concept of using tests to document requirements and drive development.
Agile testers tend to have good technical skills, know how to collaborate with others to automate tests, and are also experienced exploratory testers.
They’re willing to learn what customers do so that they can better understand the customers’ software requirements.
Traditional vs. Agile Testing
Role Testers Play
The role of the tester with agile methods is an area that has received increasing attention. Initially with a focus on unit testing and ‘acceptance’ testing it appeared the system tester did not have a role in agile.
As Cem Kaner put it:
‘The nature of the tester's role changes in iterative projects. We are no longer the high-profile victims, we are no longer the lonely advocates of quality, we are merely (!) competent service providers, collaborating with a group that wants to achieve high quality.’
Testing and Testers on Agile Projects
1. It comes as no surprise to testers that working software is not the same as code – the tester clearly needs to be involved in not only assessing the product, but in deciding how the product is to be assessed. However, with automated unit tests in the hands of the coders, and confirmation-focused acceptance testing driven by the customer, testers should be aware that they will not be the sole – or even the primary – owner of deciding what works, and what doesn’t.
2. Testers need to be able to interact directly with designers and coders to understand the technological imperatives and restrictions that affect the software and its unit tests.
3. Testing will be driven by what is important to a user, rather than to fulfill a procedural requirement. It is better to have communication between tester, customer and designer than to maintain independence of the test team. In practice, it is common to find large-scale automated unit testing on agile projects, to confirm that code works as expected. The product will be judged by the customer typically by manual, confirmatory tests, with close observation for undesirable behaviors. Testing by testers is often driven by the need to measure the system’s performance and to find surprises – tools are very much in evidence, but rigid test scripts and procedures do not give the requisite opportunity for discovery, diagnosis and exploitation.
4. Testers are key collaborators with the customer, and on some agile projects will take on much of the role of the customer in designing and executing confirmation-driven acceptance tests. However, although testers traditionally make good customer advocates, working closely with a customer is preferable to becoming a proxy. Test strategies which lean heavily on an unchanging set of requirements (for example: designing and coding tests to be bought together with code late in the project; prioritizing tests based on a fixed risk assessment; testing only what has been agreed in the contract; reporting bugs only against fixed requirements) may be considered to be fatally flawed in the light of this value. Iterative collaboration is favored over a negotiated bill of work.
While a developer is coding a task, it is impossible for a tester to test it (it doesn't exist yet). What then is the role of a tester at this point.
1. Testers can prepare their test plans, test cases, and automated tests for the user stories before (or while) they are implemented. This helps the team discover any inconsistency or ambiguity in the user stories even before the developers write any code.
2. The tester could be working with the customer to fine tune the stories in the sprint.
3. They can often be involved in designing the tests that the coder will write to perform TDD.
4. If the agile team is fairly advanced then the tester would normally be writing the ATDD (Acceptance Test Driven Development) tests. These could be in a tool such as Fitnesse or Robot Framework or they could be more advanced ruby tests or even some other programming language. Or in some cases, simple record and playback can often be beneficial for a small number of tests.
5. They would obviously be writing/planning some exploratory testing scenarios or ideas.
6. The tricky thing to comprehend sometimes for the team is that the story does not have to be complete in order to drop it to the test stack for testing. For example the coders could drop a screen with half of the fields planned on it. The tester could test this half whilst the other half is being coded and hence feedback in with early test results. Testing doesn't have to take place on "finished" stories.
Is the tester now involved in unit testing? Is this done parallel to black box testing?
Testers only test code that passes all of the automated unit, integration and acceptance tests, which are all written by the developers. This split may be different elsewhere, though; for example your testers could be writing automated acceptance tests.
What does the tester do during a sprint where primarily infrastructural changes have been made, that may only be testable in unit testing?
1. Tester’s workload will vary between sprints, but regression tests still need to be run on [if any] changes...
2. You may also find that having the testers spend the first couple of days of each sprint testing the tasks from the previous sprint may help, however it's better to get them to nail down the things that the developers are going to be working on by writing their test plans.
3. Ensure that project or sprint requirements are clear, measurable and testable. In an ideal world each requirement will have a fit criterion written down at this stage. Determine what information needs to be automatically logged to troubleshoot any defects.
4. Prepare a project specific test strategy and determine which QA steps are going to be required and at which project stages: integration, stress, compatibility, usability, performance, beta testing etc. Determine acceptable defect thresholds and work out classification system for defect severity, specify guidelines for defect reporting.
5. Specify, arrange and prepare test environment: test infrastructure and mock services as necessary; prepare test data; write scripts to quickly refresh test environment when necessary; establish processes for defect tracking, communication and resolution; prepare for recruitment or recruit users for beta, usability or acceptance testing. Write test scripts.
6. Ideally the tester would be working with the team and the customer (who by the way, is part of the team!) to define the planned stories and build in some good, detailed acceptance criteria. This is invaluable and can save loads of time later down the line. The tester could also be learning new automation techniques, planning test environments, helping to document the outcome of the planning.
Specific Technical Skills For Agile Tester’s Toolkit
1. Automation Skills
Learn how to evaluate and choose the right tools, so you can help your team create maintainable automated regression tests. You can free up time for essential testing activities such as exploratory testing.
2. Acceptance Test-driven Development
Communication skills and good domain understanding enable testers to help business experts give good examples of both desired and undesired system behavior. We can turn these examples into tests that help the programmers understand what code to write. This is called acceptance test-driven development, and it is a major step toward building quality into the code and preventing defects.
3. Learning Styles
We all have blind spots that may prevent us from learning or triggers where we shut down and don’t hear the message anymore. Keep your emotional “hot buttons” in mind and focus on what you can learn from instructors, material, or teammates to enhance your abilities. Mentors with different backgrounds or from other industries besides testing and software development might work best with your learning style. Don’t limit yourself to coaches, mentors, and instructors who work specifically in software testing.
4. Learning Resources Examples
References:
1. Stackexchange
2. Searchsoftwarequality
3. Kohl
4. mcbreen
5. newsweaver
6. Stackoverflow
7. LEARNING FOR AGILE TESTERS, Part 2 by Lisa Crispin and Janet Gregor, Better Software Magazine Sept/Oct 2011
8. AGILE TESTING A PRACTICAL GUIDE FOR TESTERS AND AGILE TEAMS by Lisa Crispin & Janet Gregory
This post contains:
1. Traditional style QA
2. Agile Approaches
3. An Agile Tester
4. Traditional vs. Agile
5. Role Testers Play
6. Testing & Testers on Agile Projects
7. Question: While a developer is coding a task, it is impossible for a tester to test it (it doesn't exist yet). What then is the role of a tester at this point.
8. Question: Is the tester now involved in unit testing? Is this done parallel to black box testing?
9. Question: What does the tester do during a sprint where primarily infrastructural changes have been made, that may only be testable in unit testing?
10. Specific Technical Skills For Agile Tester’s Toolkit
Traditional Style Quality Assurance
Agile approaches are changing the conversation about software development
An Agile Tester
A professional tester who embraces change, collaborates well with both technical and business people, and understands the concept of using tests to document requirements and drive development.
Agile testers tend to have good technical skills, know how to collaborate with others to automate tests, and are also experienced exploratory testers.
They’re willing to learn what customers do so that they can better understand the customers’ software requirements.
Traditional vs. Agile Testing
| Traditional | Agile |
|---|---|
| In the phased approach diagram (see previous image), it is clear that testing happens at the end, right before release. The diagram is idealistic, because it gives the impression there is as much time for testing as there is for coding. In many projects, this is not the case. The testing gets “squished” because coding takes longer than expected, and because teams get into a code-and-fix cycle at the end. | Agile is iterative and incremental (see previous image). This means that the testers test each increment of coding as soon as it is finished. An iteration might be as short as one week, or as long as a month. The team builds and tests a little bit of code, making sure it works correctly, and then moves on to next piece that needs to be built. Programmers never get ahead of the testers, because a story is not “done” until it has been tested. |
| Tests are usually created from a requirements document. | Rather than creating tests from a requirements document that was created by business analysts before anyone ever thought of writing a line of code, someone will need to write tests that illustrate the requirements for each story days or hours before coding begins. |
Role Testers Play
The role of the tester with agile methods is an area that has received increasing attention. Initially with a focus on unit testing and ‘acceptance’ testing it appeared the system tester did not have a role in agile.
As Cem Kaner put it:
‘The nature of the tester's role changes in iterative projects. We are no longer the high-profile victims, we are no longer the lonely advocates of quality, we are merely (!) competent service providers, collaborating with a group that wants to achieve high quality.’
Testing and Testers on Agile Projects
1. It comes as no surprise to testers that working software is not the same as code – the tester clearly needs to be involved in not only assessing the product, but in deciding how the product is to be assessed. However, with automated unit tests in the hands of the coders, and confirmation-focused acceptance testing driven by the customer, testers should be aware that they will not be the sole – or even the primary – owner of deciding what works, and what doesn’t.
2. Testers need to be able to interact directly with designers and coders to understand the technological imperatives and restrictions that affect the software and its unit tests.
3. Testing will be driven by what is important to a user, rather than to fulfill a procedural requirement. It is better to have communication between tester, customer and designer than to maintain independence of the test team. In practice, it is common to find large-scale automated unit testing on agile projects, to confirm that code works as expected. The product will be judged by the customer typically by manual, confirmatory tests, with close observation for undesirable behaviors. Testing by testers is often driven by the need to measure the system’s performance and to find surprises – tools are very much in evidence, but rigid test scripts and procedures do not give the requisite opportunity for discovery, diagnosis and exploitation.
4. Testers are key collaborators with the customer, and on some agile projects will take on much of the role of the customer in designing and executing confirmation-driven acceptance tests. However, although testers traditionally make good customer advocates, working closely with a customer is preferable to becoming a proxy. Test strategies which lean heavily on an unchanging set of requirements (for example: designing and coding tests to be bought together with code late in the project; prioritizing tests based on a fixed risk assessment; testing only what has been agreed in the contract; reporting bugs only against fixed requirements) may be considered to be fatally flawed in the light of this value. Iterative collaboration is favored over a negotiated bill of work.
While a developer is coding a task, it is impossible for a tester to test it (it doesn't exist yet). What then is the role of a tester at this point.
1. Testers can prepare their test plans, test cases, and automated tests for the user stories before (or while) they are implemented. This helps the team discover any inconsistency or ambiguity in the user stories even before the developers write any code.
2. The tester could be working with the customer to fine tune the stories in the sprint.
3. They can often be involved in designing the tests that the coder will write to perform TDD.
4. If the agile team is fairly advanced then the tester would normally be writing the ATDD (Acceptance Test Driven Development) tests. These could be in a tool such as Fitnesse or Robot Framework or they could be more advanced ruby tests or even some other programming language. Or in some cases, simple record and playback can often be beneficial for a small number of tests.
5. They would obviously be writing/planning some exploratory testing scenarios or ideas.
6. The tricky thing to comprehend sometimes for the team is that the story does not have to be complete in order to drop it to the test stack for testing. For example the coders could drop a screen with half of the fields planned on it. The tester could test this half whilst the other half is being coded and hence feedback in with early test results. Testing doesn't have to take place on "finished" stories.
Is the tester now involved in unit testing? Is this done parallel to black box testing?
Testers only test code that passes all of the automated unit, integration and acceptance tests, which are all written by the developers. This split may be different elsewhere, though; for example your testers could be writing automated acceptance tests.
What does the tester do during a sprint where primarily infrastructural changes have been made, that may only be testable in unit testing?
1. Tester’s workload will vary between sprints, but regression tests still need to be run on [if any] changes...
2. You may also find that having the testers spend the first couple of days of each sprint testing the tasks from the previous sprint may help, however it's better to get them to nail down the things that the developers are going to be working on by writing their test plans.
3. Ensure that project or sprint requirements are clear, measurable and testable. In an ideal world each requirement will have a fit criterion written down at this stage. Determine what information needs to be automatically logged to troubleshoot any defects.
4. Prepare a project specific test strategy and determine which QA steps are going to be required and at which project stages: integration, stress, compatibility, usability, performance, beta testing etc. Determine acceptable defect thresholds and work out classification system for defect severity, specify guidelines for defect reporting.
5. Specify, arrange and prepare test environment: test infrastructure and mock services as necessary; prepare test data; write scripts to quickly refresh test environment when necessary; establish processes for defect tracking, communication and resolution; prepare for recruitment or recruit users for beta, usability or acceptance testing. Write test scripts.
6. Ideally the tester would be working with the team and the customer (who by the way, is part of the team!) to define the planned stories and build in some good, detailed acceptance criteria. This is invaluable and can save loads of time later down the line. The tester could also be learning new automation techniques, planning test environments, helping to document the outcome of the planning.
Specific Technical Skills For Agile Tester’s Toolkit
1. Automation Skills
Learn how to evaluate and choose the right tools, so you can help your team create maintainable automated regression tests. You can free up time for essential testing activities such as exploratory testing.
2. Acceptance Test-driven Development
Communication skills and good domain understanding enable testers to help business experts give good examples of both desired and undesired system behavior. We can turn these examples into tests that help the programmers understand what code to write. This is called acceptance test-driven development, and it is a major step toward building quality into the code and preventing defects.
3. Learning Styles
We all have blind spots that may prevent us from learning or triggers where we shut down and don’t hear the message anymore. Keep your emotional “hot buttons” in mind and focus on what you can learn from instructors, material, or teammates to enhance your abilities. Mentors with different backgrounds or from other industries besides testing and software development might work best with your learning style. Don’t limit yourself to coaches, mentors, and instructors who work specifically in software testing.
4. Learning Resources Examples
References:
1. Stackexchange
2. Searchsoftwarequality
3. Kohl
4. mcbreen
5. newsweaver
6. Stackoverflow
7. LEARNING FOR AGILE TESTERS, Part 2 by Lisa Crispin and Janet Gregor, Better Software Magazine Sept/Oct 2011
8. AGILE TESTING A PRACTICAL GUIDE FOR TESTERS AND AGILE TEAMS by Lisa Crispin & Janet Gregory