QA doc by sri sri

How to Really

Prevent and Manage Bugs

in Agile Projects

Michael Dubakov TargetProcess, Founder

Table of content Zero Defects Mentality

Bug Source #1: Unclear User Stories

Clarification meeting Stories should be ready before iteration start

Bug Source #2: Manual Testing in Short Iterations

Bug Source #3: Automated Testing in Short Iterations Bug Source #4: Separate Testing Team

Bug Source #5: Bug Clusters Agile Bug Management

Bugs found in iteration Bugs found in production Bug estimation

Agile Bug Management Anti-Patterns

10 12 13

Bug fixing phase inside iteration Bug fixing iteration Skip “Zero open bugs” rule in user story’s Definition of Done

Defect Tracking Systems

Co-located Distributed

References

Foreword Today, Agile is a widely adopted software development approach. Most developers have either heard of or successfully applied popular agile practices like TDD, CI, PP, etc. But what about testers? It seems that many teams are having difficulties trying to integrate quality control teams into an agile development process. Moreover, it also seems that many teams are having difficulties with overall bug management. How to treat bugs? Does a bug have a business value? Should we estimate bugs in points or in hours? Should we treat bugs as user stories and add to velocity? This article not only answers the questions above and explains how to manage bugs in agile projects, but it also shows how not to manage bugs.

Zero Defects Mentality Are you familiar with a zero defects mentality? It looks very good from the first sight: “I’d like to have zero defects in my projects.” So, what is a zero defects mentality? Here is the quote from the book Lean Software Development: an Agile Toolkit: “One of the fastest ways to kill motivation is what is called in the US Army a zero defects mentality. A zero defects mentality is an atmosphere that tolerates absolutely no mistakes; perfection is required down to the smallest detail. The army considers a zero defects mentality to be a serious leadership problem, because it kills the initiative necessary for success on a battlefield” Mary & Tom Poppendieck Obviously, a zero defects mentality is not something that HOUSE M.D. likes ;) Moreover, I think Dr. House hates it, because it has several unpleasant effects: Not enough courage to refactor a complex, messy, buggy, but important piece of code. Inability to make an important right decision. The result is a less risky, but wrong decision. Doing everything to avoid responsibility leads to coward and stupid behavior. Zero defects may sound good. But in reality you still have errors after production. Even in predictable and (quite) easily testable industries (hardware, automobile, etc.) there are problems with 100,000 power adapters that should be replaced (or hard drive problems, or accumulators problems, I bet you can continue the list). How can we expect zero defects in software development? It is harder to test, harder to define in details, harder to predict. Software development is a complex adaptive system, and we can't predict all effects. So bugs in production are a normal thing, and by “normal” I mean that we can't bring them to zero. We can (and should) minimize them using all possible ways. Bugs are a form of waste. We should do our best to reduce bugs in software projects and eradicate waste. There are several strategies that will help: Test Driven Development. A nice side effect of TDD is a unit tests suite. You have tests for all new code, and you have unit tests for all old code. If you have good tests - you have less bugs.

Continuous integration. Instant feedback helps to identify problems early and fix them early. It saves money and you have less bugs in production. Automated regression functional testing suite. Unit tests are good, but you need something else. Functional tests emulate user behavior and identify user interface errors, integration errors, etc. Needles to say you should have continuous integration in place to succeed with automated functional tests. Root cause analysis. There are several ways to fix the bug. You may just hack the code and apply a patch (please, don’t do it at home!). You may think deeper and fix the bug nicely. Also you may look into the root of the problem and resolve it. Yes, it may take time, but you will prevent future bugs from this domain. High development skills. Ironically, high skills do not always reduce a bug rate in production. “Stars” may be strong in some areas (business layer), but may not polish things. It may lead to many small bugs, especially on UI. “Stop worrying about the defect database and start worrying about why you are still creating code where defects are discovered a significant amount of time after the code has been written.” Mary Poppendieck

Bug Source #1: Unclear User Stories I think unclear specifications are a #1 source of bugs. If you deliver something that was not intended, there will be many complaints from customers and in their opinion almost all of these complaints are bugs. Also, without instant communication it is very easy to miss some details and release an unpolished user story with many small glitches. Agile development is very rapid. In many cases the path from an idea to user story in progress is extremely short; it may be even 1 day. In this environment it is very easy to make mistakes in a user story description. Product Owner may miss some important details and as a result the story does something wrong, which was not expected. “This is the first question you must ask... is it a bug, or is it a change? I've seen a lot of bugs that have come up that were "We asked you for x, never thinking about y, so could you please change the system so that y is covered?" It's a business scenario, so there's little reason to expect a dev or tester to anticipate/test it. I know where we've struggled is in injecting items into the product backlog, we tend to classify it as a bug and so we end up with a lot of bugs, but not a lot of change in the back log. That's exactly what I think we're supposed to be avoiding in Scrum or any other agile methodology. As we see change, we should be taking it on as a feature and prioritizing it accordingly. Only when it's something broken should it be called a bug.” Jim Leonardo The best solution to this problem is communication. 5

Clarification meeting The important thing in software development is to put everyone on the same page. Developer, Tester and Product Owner should conduct a small meeting about a user story, thus forming a mini-team. If the user story has several developers and several testers, all should participate. The goals of the meeting are: To ensure that everyone understands what should be implemented. To brainstorm acceptance tests/test cases and create a check list. To reveal and eliminate all discrepancies in a story description/specification.

Product Owner

Discuss User Stories, brainstorm acceptance tests

Developer

Tester

We have these meetings at TargetProcess and they are very efficient and helpful. The side effect of the meeting is that Product Owner receives less questions about functionality later, thus having less interruptions (which is good).

Stories should be ready before iteration start It is a good idea to have all user stories described in minute detail before iteration start. Thus, testers and developers will have time to review them and prepare questions for the iteration meeting. It may drive a much better discussion and reveal some interesting problems that were unclear initially. 6

There is a danger to document more user stories than is required, and it will be a form of waste. Product Owner should maintain good balance here.

Bug Source #2: Manual Testing in Short Iterations Most testers are having hard time switching from waterfall to iterative development. Short iterations put pressure on a testing activity. It is required to write test cases fast, execute them fast and provide feedback fast. It is far away from a huge testing phase in waterfall. Under pressure testers make more mistakes, think less and miss some test cases in a hurry. Testers tend to write brief test cases and skip unusual or not obvious cases. Naturally, it is a straight way to bugs in production. The most common error is a testing phase in every iteration. It is a kind of mini-waterfall and should be avoided. Each feature should have a separate flow with its own testing phase. It is required to test by feature. Tester should start testing immediately after a feature is developed (or even earlier if possible).

Bug Source #3: Automated Testing in Short Iterations Test Driven Development is a great bug catcher. If you apply it right, you will improve product quality significantly and bugs in production will be quite rare. However, it is not as easy as it sounds. There are two levels of TDD: Unit Tests and Functional Tests. It is relatively easy to adopt unit tests and make it a mandatory practice in the development team. Developers write unit tests, then write code, then refactor. Functional tests are harder to create and maintain. And it is even harder to create them before feature implementation. And it is especially hard to create automated functional tests upfront on the User Interface level. Tools like Fitnesse simplify functional testing of business logic, but still you want to have complete UI tests using Selenium, WinRunner or any other GUI testing tool. Here are several pieces of advice that will help you catch more bugs during iterations using automated tests: Adopt TDD fully on the unit tests level. Hire highly experienced automated testers (almost like developers in skills) who can write automated tests very fast. Speed is important here. An alternative is to force developers to write automated functional tests. Create automated functional tests for the previous iteration at the beginning of the next iteration. It will be regression testing, but it is definitely better than nothing.

Bug Source #4: Separate Testing Team It is common for a waterfall process to have separate functional teams: Designers, Developers, Testers, DBA, etc. Functional teams set additional communication barriers, and poor communication leads to many problems like: Missing test cases. Different understanding of a user story by Developer and Tester. Delay between story implementation and testing phases. Obviously, these problems lead to more bugs found in production. The best thing to do here is to create a cross-functional team and put testers and developers into one shared space. It should be one team, not several sub-teams. Quite a common situation with functional departments looks like this. You have several testers assigned to your project. Then suddenly two of them get relocated to another project and replaced by two other testers. That is a real danger to your project. So, your goal is a single cross-functional team. If it is not possible, youâ&#x20AC;&#x2122;ll have to play political games. At least you should try to extract a dedicated testing team for the project and minimize the power of Functional Testing Department Manager. In this case you have a chance to join two functional teams into one.

Bug Source #5: Bug Clusters Bugs like to live in communes in a single nest. Most likely you have modules with perfect conditions for bug survival and reproduction. Bugs prefer the following places: Code written by junior developers. Code written under time pressure. Legacy (initially beautiful) code that was passed to the offshore team, then back to the main development team, then back offshore and then shared between several teams. Code that somehow does not have unit tests (what a shame!). UI (yes, most user interfaces are buggy, since developers do not unit test UI). Quite a large and complex piece of code (there is a high chance that complex logic will have some missed test cases as it is hard not to forget something). Code written in JavaScript. Integration with a third-party system. So if you have a complex UI component written by Junior Developer in JavaScript with no unit tests and under pressure - I bet you have several bugs in this code.

Anyway, bugs are nesting and so a good strategy is to find the nest and destroy it (refactor, rewrite, add unit tests, etc). Lisa Crispin has something to say about it. â&#x20AC;&#x153;If you already have a defect database, consider mining it for information about where defects cluster, and focus on cleaning up the related code.â&#x20AC;? - Lisa Crispin http://home.att.net/~lisa.crispin/CrispinFinal.pdf

Software

Bug

Authentication module

Bug

Bug Bug Bug Bug

Content module Bug Bug

Bug

Bug Bug

Bug

Reporting module

Bug

If you have a defect tracking system, you may use tags, thus categorizing defects. It will be quite easy to find problematic areas in your software. For example, you may find out that most bugs live in the Reports or Email Integration areas.

Agile Bug Management Agile development books often skip a bug management practice. Many agile people assume that the problem is obvious, unfortunately it is not the case. In general, there are two types of bugs: bugs found during iteration development and bugs found in production. Strategies are quite different as you may expect.

Bugs found in iteration There is a huge difference between a bug found during story development and a bug found by the customer in production. A bug found during story development is much easier to fix, since you have focus on the story, you know what you did 1-3 days ago and everything is fresh in 9

mind. You may even recall some places in the code where the bug can live right away. If a bug is found a month from now, it will take time to switch to it, fix, include into the new build and deliver. So, how should we handle such bugs? First, do not estimate bugs during story development at all. Second, do not prioritize them. Bugs are a usual part of the development process and on average in each iteration you will have quite similar time spent on bug fixing. So why bother with estimation and prioritization? One simple rule replaces them: all bugs found in the iteration should be fixed. No exceptions. No excuses. If you do not estimate bugs, they do not affect iteration velocity. If you use the same estimate technique during several iterations, most likely you will have similar bug rates. It means that the bug fixing deviation does not affect velocity.

Iteration #1

User Story

Bug Bug

User Story

Bug

User Story

Bug

Time

Bug

Bug Bug

User Story Bug

Bug

Bug Bug Bug

Bug

0 open bugs

Alternatively you may estimate bugs as tasks (in hours), but I donâ&#x20AC;&#x2122;t see much value it this effort.

Bugs found in production Unfortunately, we have to deal with bugs in production. No software is bug free. There are bugs in business software, medical software and even space shuttle software. Sure medical software has less bugs, but they still exist (http://www.cs.tau.ac.il/~nachumd/ verify/horror.html).

Bugs as User Stories One strategy is to put bugs into the product backlog and treat them as user stories. From the customerâ&#x20AC;&#x2122;s perspective there is no much difference between bug in production and a new feature request. The customer wants to have a problem solved, and the problem may be expressed as a bug or may be expressed as a user story.

Iteration #1 User Story

Production Bug

Bug Bug

User Story

Bug

Bug Bug

Production Bug

Time

Bug Development Team

Is this bug with login screen more important than the “Advanced Search” user story? It is a subject for Product Owner to decide. He should maintain a single backlog with bugs and user stories prioritized and select bugs and stories for the next iteration. In such a scenario you will have to estimate bugs in points or in hours to have a fair prediction how many bugs and stories you may take and implement in the next iteration. It sounds somewhat counterintuitive, since bugs are a form of waste. But still prioritization and estimation are important in this scenario. However there is a danger in this approach: If defects are viewed as features with negative value, they become managed as if they were features. Development groups store up prioritized repositories of bugs, treat bugs as user stories, outsource the fixing of bugs, and so on. While each of those can be a useful technique or perspective in dealing with a project in transition or crisis, it is not a longterm view that should be encouraged. After all, as the "Manifesto for Agile Software Development" says, "Working software is the primary measure of progress." It is a little disingenuous to pass off a feature with known defects as complete and working — "Yes, it's done... but there are a few bugs." - Kevlin Henney

People dedicated to bug fixing Another strategy is to have dedicated people for bug fixing. For example, you may dedicate about 10-20% of a development team’s effort to production bug fixes. Alternatively, you may have one developer each day on production bug fixing, and developers may rotate daily.

Iteration #1

User Story Bug Bug

User Story Bug

Bug

User Story

Bug

Time Bug

Bug

Development Team In this case, 25% of the development team is solely dedicated to fixing bugs found in production (1 of 4 developers) Production Bug

Production Bug

Support Developer

Free time. Support Developer drinks bear or joins the development team (and enjoys user stories). But if a new bug is found in production, she switches to it with no delays.

Using this approach, you have two separate flows: one for user stories and another for bugs found in production. Obviously, each flow should be managed separately. While for user stories you have usual metrics like an iteration burn down chart, for a bug flow you need to invent something different. If you do not estimate bugs, all you have is the bugs fixed/day metric. If you do estimate them, you may create a bug burn up chart or points fixed/day chart. If you have few bugs, good advice will be to not track the bug fixing flow at all. But if you have many bugs that will take several weeks to fix, you need some metrics to forecast when you will be able to fix all Critical bugs, all Normal bugs, all Small bugs, etc. Also you should know how many new bugs are coming from production each week. It will definitely influence forecasts. Weeks to a bug free release = Total bugs / (Weekly bug fixing rate - Weekly new bugs rate) For example, you have 40 bugs to fix, weekly bug fixing rate is 10 bugs/week, and each week 2new bugs are found. In this case Weeks to a bug free release = 40 / (10 - 2) = 5 weeks.

Bug Estimation You may not estimate bugs. There are several reasons not to estimate: Often bugs are hard to estimate. Even a small problem may take a day to fix (unexpectedly revealing dirty roots that should be cleared up). Bugs should be fixed with no exceptions. If you have few bugs from production and use dedicated people for bug fixing, estimation does not bring any additional value. Bug estimation is required when: You have many bugs in production and want to provide some forecasting. You treat bugs as user stories and plan iteration accordingly. 12

Agile Bug Management Anti-Patterns There are several dangerous misunderstandings (anti-patterns) related to bug management that are quite common. It is better to recognize the problem early and fight to win. Here are three common bug management anti-patterns in a development process.

Bug fixing phase inside iteration Symptom: You have several days at the end of an iteration fully dedicated to bug fixing. Iteration #1

Implement User Stories User Story

Bug Fixing

User Story

Time

Bug

If you have a bug fixing phase inside an iteration, you have mini-waterfall in each iteration, which is not a very good idea (in agile we blame waterfall, don’t we?). A story should be tested as soon as it is completed.

Bug fixing iteration Symptom: You run several iterations with almost no or little bug fixing and then have one (or even two) iterations that are fully dedicated to bug fixing. Release #1

Iteration #1 Implementing stories User Story

User Story

Iteration #2 Implementing stories User Story

User Story

Time

Iteration #3 Bug fixing Bug

Bug

A bug fixing iteration kills your iterative development. It is even worse than mini-waterfall. Such iterations increase your work in progress dramatically. You have nothing done during several iterations before a bug fixing iteration. You have nothing to ship. Moreover, bug fixing iterations reduce motivation, people do not like to fix bugs during 2 weeks or a full month.

Skip “zero open bugs” rule in user story’s Definition of Done Symptom: You have open bugs in a completed iteration. You may have decided that some bugs discovered during story development are unimportant and may be postponed to future releases/iteration. That is a very dangerous practice that leads to bug accumulation. Moreover, it may be a reason for a bug fixing iteration (see above). The best strategy is to have a “zero open bugs” rule in DoD for a user story.

Defect Tracking Systems Should we use a defect tracking system? The answer is “it depends”. Here are all possible alternatives: Use a defect tracking system (excel, web based, desktop). Use Task Board. Write bugs on sticky notes and stick them to Task Board. Use index cards. Write bugs on a card and give it to Developer. Personal communication (use voice and memory). Just tell Developer about the problem. Let’s see under what conditions you may use these approaches. Co-located

Distributed

Small team

Task Board, index cards, personal communication

Defect tracking system

Medium Team

Task Board, defect tracking system

Defect tracking system

Large Team

Defect tracking system

Obviously, a defect tracking system is a must for a distributed team. A distributed team demands more formal communication and a single place to store all bugs. A small co-located team may use any approach that fits their process. Medium and large colocated teams can’t rely on personal communication and simple story cards. Task Board may work for a medium team, but a large team most likely needs a defect tracking system. I was surprised that our programmers find the defect-tracking system quite useful. The ability to read clear, detailed steps to reproduce the problem saves them time. - Lisa Crispin

References Coping with Bugs on an Agile/Scrum Project http://www.infoq.com/news/2009/07/coping-with-bugs Software testing is an activity, not a phase or a department http://searchsoftwarequality.techtarget.com/news/article/ 0,289142,sid92_gci1337882,00.html Fix defects right away http://www.cyphersec.com/2009/05/21/fix-bugs-now/