Problem statement:
All software development teams (whether Agile or not) entering the maintenance phase are charged with keeping application/code quality high, while rolling out frequent incremental updates. Now unit test coverage (nUnit, mbUnit, phpUnit etc.) and automated functional test coverage (viz. Selenium, QTP, Win Runner etc.) can ease this burden. However, it is not feasible to have 100% test coverage and then to keep these tests updated ongoing at a high coverage rate. A savvy Dev Lead will use instinct and experience to know how to best deploy Dev and QA resources while minimizing risk. (Check back for our follow up article on criteria to optimize test automation)
At the same time, it is also cost prohibitive to run every possible manual test case every time regardless of scope of the release across multiple environments. Manual tests are cheap to write/maintain but expensive to execute repeatedly. (This is, of course, the exact reverse of the cost/benefit proposition for automated tests.) Given that there will always be some manual testing involved in a release, technical leads must decide on the scope of testing for every release. Decision-making of what the scope is for the dev team is often a careful, deliberate, and explicit process. But when it comes to the scope of testing though, very often QA leads decide this implicitly based on the items addressed in a release e.g. if a bug in Feature X was addressed then Feature X needs be tested. Most times that works well if you have experienced QA leads. Every now and then though, the code fix for Feature X may be made in a component A that is reused in Feature Y. Throw in a minor oversight and… Voilà! You have a bug introduced in Feature Y that no one thought to test with a clear path to production.
So what is the solution?
Enter the Change Map. This is nothing more than a simple Traceability Matrix that explicitly identifies/references the test cases to be executed for a change request, typically in the context of a ticketing system.
Here is how it works: When a new change request (or bug) is approved for development, a Change Map should be identified. This is done through a conversation between Functional owner, Solution Lead, developer(s), QA lead, and testers to identify what are the areas/modules which will get affected by this change. It should be done first when the desired functionality is discussed, and then revalidated when the technical design approach has solidified.
Not sure how it would work? Let’s play “What was my team thinking?!”
Let us consider this interaction occurring in the context of a role based web application that is currently in production.
Client (Day 1): “We want to add a new role that has access to XYZ. Should be a simple change right?”
Business Analyst (Day 1): “Yes, our application already supports role based access. This change is straight forward for implementation.”
Tech Lead (Day 2): “Hmm… one of the areas we are restricting wasn’t considered a separate restricted resource before. We may need to update the base class to recognize this sub-feature as a separate feature”
Developer (Day 2): “Yeah, I know where to do that. Done.”
QA Lead (Day 3): ”We need to test this new role thoroughly”
Tester (Day 3): “New role works fine!”
PM (Day 4): “We are a GO for the release.”
Client (Day 5): “My call center is deluged with existing users unable to use the system! You just broke everything, I thought you said this was an easy change?!”
In the above scenario, the change to add a role introduced a bug into a pre-existing role and the bug had a clear path to production since no one brought up that other roles need to be tested. Here is what could have been different if we had used the Change Map:
Client (Day 1): “We want to add a new role that has access to XYZ. Should be a simple change right?”
Business Analyst (Day 1): “Yes, our application already supports role based access. This change should be straight forward for implementation but I will get back to you once my Tech Lead and Tester review it.”
Tester (Day 2): “Well, I have my test cases drawn up for the new role. That part is easy, but Mr. Tech Lead can advise if anything else needs to be added to the Change Map”
Tech Lead (Day 2): “Hmm… one of the areas we are restricting wasn’t considered a separate restricted resource before. We may need to update the base class to recognize this sub-feature as a separate feature. Ms. Developer, can you validate that and if necessary add those to the Change Map”
Developer (Day 2): “Yeah, I know where to make that change. And I’ve updated the ticket with the additional Change Map test cases.”
QA Lead (Day 3):”We need to test this new role thoroughly, as well as the additional test cases called out in the Change Map”
Tester (Day 3): “New role works fine but older roles are broken!”
Developer (Day 3): “Ah! Missed a spot, there you go”
PM (Day 4): “We are a GO for the release.”
Client (Day 5): “New role works fine. Here is another change I need…”
How does this help?
1. Codify decision making criteria: The main value here was to make the implicit process of deciding what gets tested explicit and involve non-QA actors in the decision making process
2. Improve communication/cross role understanding: Some minor tweaks to the mental framework of team members to think as a team and hold each other responsible for considering downstream effects of code changes can reap significant benefits
3. Continuous process improvement: If bugs still sneak through, perform an After Action Review to determine what updates need to be made to #1 to prevent recurrence
NOTE: Change Map should be a light weight addition to the ticketing process. When in doubt, players should be looking to broaden testing scope rather than narrow it.
Happy Change Mapping!
