stp-2006-12 by Morten Kristiansen

Publication

VOLUME 3 • ISSUE 12 • DECEMBER 2006 • $8.95

: ST ES BE CTIC ation nt A r e PR figu gem n a Co an M

www.stpmag.com

Automation SDLC Part 2: Celebrate With Winning Candidates

The Envelope, Please 2006 Tester’s Choice Awards Mercury Pops the Cork For Second Year Running Role With It— The Basics Of Testing in Context How To Make Requirements-Based Testing Easy as 1-2-3

Ship Software OnTime.

™

" / ÓääÈ The Fast & Scalable Team Solution for... Defect & Issue Tracking • Feature & Change Tracking • Task & To-do List Tracking • Helpdesk Ticket Tracking OnTime is the market-leading project, defect and feature management tool for agile software development and test teams. OnTime facilitates tracking, analyzing and trending team-based software development efforts in an intuitive and powerful user interface. A fully customizable UI, powerful workflow, process enforcements, two-way email communications and custom reports combine to help software development teams ship software on-time!

Available for Windows, Web & VS.NET 2003/2005 OnTime 2006 Professional Edition

OnTime 2006 Small Team Edition

• For Teams of 10 to 1,000 Members • From $149 Per User

• • • •

For Teams up to 10 Members Free Single-User Installations $495 for 5-Team Members $995 for 10-Team Members

800·653·0024 software for software development™

www.axosoft.com

Only $495 for up to 5 Users • Only $995 for up to 10 Users Free Single-User Installations

L ?K@A NAREAS

S=O O?=J ?L

?DA?GA@ EJ

REAS NA=@U BKN NA

EO LNEJP ?LL

DA >QEH@

NA=@U BKN P

=NA =HH BEHAO

?DA?GA@ EJ

EJC AHH IA =JU PD ?=J P UKQ P

AHOA

?DA?GA@ EJ

â&#x20AC;&#x153;Checked inâ&#x20AC;? tells you nothing. Your configuration management tool should tell you more.

New Surround SCM 5 does.

Seapine

5 Software Configuration Management ÂŽ

Surround SCM

Seapine ALM Solutions: TestTrack Pro Issue & Defect Management TestTrack TCM Test Case Planning & Tracking Surround SCM Configuration Management QA Wizard Automated Functional Testing

New Surround SCM 5.0 with change automation and custom metadata gives you more control over your software change process and a clearer view of the state of files.

â&#x20AC;˘ Create workflows to ensure changes to files follow your processes.

Know whether a file you are including in the build has been code reviewed. Ensure design documents have been through the review process. Control who can make changes to files after they have been approved.

â&#x20AC;˘ Automate the development process with powerful in-application triggers.

Surround SCM 5.0â&#x20AC;&#x2122;s virtual branches and collaboration features help streamline parallel development. Thereâ&#x20AC;&#x2122;s never been a better time to change to Surround SCMâ&#x20AC;&#x201D;the next evolution in SCM.

â&#x20AC;˘ Integrate seamlessly with TestTrack and other popular development and build tools.

â&#x20AC;˘ See beyond â&#x20AC;&#x153;Checked inâ&#x20AC;? to know the true status of files in the change process.

â&#x20AC;˘ Improve security with single sign-on support.

â&#x20AC;˘ Streamline communication with automated change notifications and flexible reporting.

Download your FREE fully functional evaluation software now at www.seapine.com/stpscm5 or call 1-888-683-6456.

VOLUME 3 • ISSUE 12 • DECEMBER 2006

Contents

Publication

COV ER STORY

The Envelope, Please

Our Second Annual Tester’s Choice Awards honored some old favorites and some exciting up-and-comers. By Edward J. Correia and Laurie O’Connell

An Automation Vote of Conf idence

Taking the plunge into automation? In this second article in a series of three, explore a range of selection criteria and some common-sense guidelines. By Bob Galen

Depar t ments

RequirementsBased Testing Ideas

7 • Editorial This year’s STPCon proved a simple truth: Good conferences attract good people.

Your clients’ needs are first and foremost: To fulfill them, RBT can be coupled with existing QA and testing best practices. By Moty Aharonovitz

9 • Out of the Box

New products for developers and testers.

Align Testing With Its Context

12 • Peak Performance

As testers, we manage confidence, report on risk—and ask questions to create tests that resonate with context.

35 • Best Practices

What does it really mean to be a professional? By Scott Barber

SCM is more than just code check-in and check-out. By Geoff Koch

By Adam White and Steve Rabin

38 • Future Test Model-driven architecture is a tool for By Wolfgang Kaml the future.

DECEMBER 2006

www.stpmag.com •

A BZ Media Event

2007

April 17-19, 2007 San Mateo Marriott San Mateo, CA www.stpcon.com

Mark Your Calendar! After selling out the last two East Coast STPCons, the Software Test & Performance Conference is adding a West Coast event.

Get ready to saddle up and choose from more than 60 great classes!

Ed Notes VOLUME 3 • ISSUE 12 • DECEMBER 2006 EDITORIAL Editor Edward J. Correia +1-631-421-4158 x100 ecorreia@bzmedia.com Managing Editor Patricia Sarica psarica@bzmedia.com Copy Editor Laurie O’Connell loconnell@bzmedia.com

Contributing Editors Scott Barber sbarber@perftestplus.com Geoff Koch koch.geoff@gmail.com Editorial Director Alan Zeichick +1-650-359-4763 alan@bzmedia.com

ART & PRODUCTION Art Director LuAnn T. Palazzo lpalazzo@bzmedia.com

Art /Production Assistant Erin Broadhurst ebroadhurst@bzmedia.com

SALES & MARKETING Publisher

List Services

Ted Bahr +1-631-421-4158 x101 ted@bzmedia.com

Nyla Moshlak +1-631-421-4158 x124 nmoshlak@bzmedia.com

Advertising Sales Manager

Reprints

David Karp +1-631-421-4158 x102 dkarp@bzmedia.com

Lisa Abelson +1-516-379-7097 labelson@bzmedia.com

Advertising Traffic

Accounting

Phyllis Oakes +1-631-421-4158 x115 poakes@bzmedia.com

Viena Isaray +1-631-421-4158 x110 visaray@bzmedia.com

Marketing Manager

Marilyn Daly +1-631-421-4158 x118 mdaly@bzmedia.com READER SERVICE Director of Circulation

Agnes Vanek +1-631-421-4158 x111 avanek@bzmedia.com

Customer Service/ Subscriptions

+1-847-763-9692 stpmag@halldata.com

Cover Photograph by José Carlos Pires Pereira

President Ted Bahr Executive Vice President Alan Zeichick

BZ Media LLC 7 High Street, Suite 407 Huntington, NY 11743 +1-631-421-4158 fax +1-631-421-4130 www.bzmedia.com info@bzmedia.com

Software Test & Performance (ISSN- #1548-3460) is published monthly by BZ Media LLC, 7 High St. Suite 407, Huntington, NY, 11743. Periodicals postage paid at Huntington, NY and additional offices. Software Test & Performance is a registered trademark of BZ Media LLC. All contents copyrighted 2006 BZ Media LLC. All rights reserved. The price of a one year subscription is US $49.95, $69.95 in Canada, $99.95 elsewhere. POSTMASTER: Send change of addresses to Software Test & Performance, PO Box 2169, Skokie, IL 60076. Software Test & Performance Subscribers Services may be reached at stpmag@halldata.com or by calling 1-847-763-9692.

DECEMBER 2006

Testers Are Still High on Mercury single view. “No other tool In our annual Tester’s gives you one screen for all Choice awards, Mercury results,” Mar said. Interactive has taken the While I’m on the subject Grand Prize for the second of STPCon, and at the risk year running. of sounding completely This year, testers chose biased (which I am), I’d like QuickTest Professional, Merto thank those of you in cury’s functional and regresattendance. This year’s consion test automation tool, ference was the best ever. which also topped three Sure, I work for BZ additional categories. In Media—the company that 2005, the company was recEdward J. Correia produces STPCon—and ognized for TestDirector for of course I want the conference to do Quality Center. This year, Mercury really well. In fact, it’s my job to help attract cleaned up, with eight additional top good speakers, to boost attendance, prizes and three finishes as runner-up. and to make sure the show continues TestDirector also did well this year, topto thrive. Therefore, I’m the least credping the Test/QA Management and ible person to be making the grandiose Defect/Issue Management categories. claim that its triennial occurrence was Another Mercury product popular the superior one. among testers is LoadRunner, which has But don’t take my word for it. Listen to spawned at least one independent conthe many attendees who told me how sultancy dedicated to training in its use great the speakers were and the speakers and helping organizations to take advanwho spoke of the great attendees. “This is tage of its full capabilities. the only conference I speak at where the At the Software Test & Performance audience is truly engaged,” said one Conference in Boston last month, I had speaker to me, a sentiment echoed in difthe pleasure of meeting Wilson Mar, a ferent words by others. man who appears to have devoted much While this was my first STPCon, which of his time to educating the world on the easily could evoke a favorable opinion, it virtues of LoadRunner (www.wilsonmar was not my first conference. Nor was it my .com)—and making a living at it. In the first catering to a small group focused on few short minutes that we spoke, he software development practices—which helped me understand why the tool can and should become a vital link in every brings me to the catering. What confersoftware development life cycle. ence has ever served better barbequed Mar said that above all else, salmon, richer, more plentiful coffee (not LoadRunner’s analysis capabilities are a to mention that it was Starbucks) and central component of its value to the tastier chocolate-chip cookies, along with enterprise. “It can grab multiple files and fruits, pastries and other bountiful goodcompare metrics from several runs,” he ies? All included with your registration said. This enables testers to track their fee! None, I dare say. progress, he explained, adding, “You So here’s to STPCon, the undisputed get total control over what goes on,” leader of conferences for the software referring to LoadRunner’s ability to contester. And there’s one more way I know figure multiple sets of test parameters April’s STPCon in California will be and loads, monitor performance of all even better: For my keynote introducinvolved entities, and analyze and repeat tions, I’ll have five months to come up the tests. Results of all tests are visible in a with funnier jokes. ý www.stpmag.com •

CodePro AnalytiX™ is developed by the experts who brought you the popular book Eclipse: Building Commercial Quality Plugins — Eric Clayberg & Dan Rubel

CodePro AnalytiX for Eclipse, Rational® and WebSphere®

™

Drive out quality problems earlier in the development process. Powerful code audits and metrics make every developer a Java expert.

Create tight, maintainable systems. Wizards generate unit and regression tests, and analyze testing code coverage.

Enforce quality measures across teams. Define, distribute, and enforce corporate coding standards and quality measures, across individuals and teams… no matter where they’re located.

Key Features of CodePro AnalytiX Y Detect & correct code quality issues... automatically Y Define, distribute & enforce quality standards across development teams Y 800+ audit rules and metrics, 350+ Quick Fixes Y Powerful management reporting Y Code metrics with drilldown & triggers Y Audit Java, JSP and XML files Y JUnit test case generation

Spend less time and money to develop high-performance Java systems.

Y Code coverage analysis of test cases

Dramatically improve your application development process and code quality… leading to more reliable, maintainable systems.

Y Integrated team collaboration

the development tools experts

www.instantiations.com 1-800-808-3737 No one outside of IBM has more experience creating Eclipse-based Java development tools

™

Y Dependency analysis & reporting Y Javadoc analysis & repair Y Seamless integration with Eclipse, Rational® and WebSphere® Studio

Out of t he Box

Klocwork K7.5 Reaches Beyond Security Static Code Analysis Klocwork has implemented runtime and multicore rules in K7.5, taking it beyond its original scope as a security vulnerability checker. The static code analysis tool for C/C++ and Java was updated in late October; pricing remains at US$2,995 per user. According to Ian Gordon, vice president of product management at Klocwork, the tool can now be used to locate potential performance issues or those that could lead to crashes. K7.5 also now includes 66 new Java code–checker rules for detecting runtime errors that he said could “cause real problems, like failing to close a resource.” For C/C++ applications that might be running on multicore systems, K7.5 introduces what Gordon said was the first in a series of analysis rules that check for locks and unlocks in code to help prevent deadlock conditions common to such systems. “You want to make sure that one chunk of code completes running on one core before it get swapped to another core,” Gordon said.

“Until an unlock is executed, threads are not released. This could cause significant performance issues,” he added. Gordon said K7.5 also now includes better integration with Apache’s Ant build tool. “We now effectively have a tool that watches your Ant build and tells us how to do our analysis.” Previous integration with Ant was manual. “This is now hands-off; there’s no

iTKO Is Setting LISA Free SOA test tools maker iTKO recently released LISA WS-Testing, a free version of LISA 3.5 Complete SOA Testing Platform, with functionality equal to the single-user version of that tool. The free version, available now at www.itko.com/ws-testing, is a fully functional test solution that QA teams can use to test their Web services deployments without the need for any special coding, the company claimed in a news release. The tool also is useful for unit, functional and regression tests needed to build and launch against current WSDL and SOAP objects. “This is a commercial product, not trialware or an open source utility,” said a statement from John Michelsen, DECEMBER 2006

iTKO co-founder and chief architect. He also claimed that the tool meets or exceeds the capabilities of its commercial competitors. iTKO is offering a free perpetual license and one year of upgrades to the first 50,000 people who download the tool. “Everyone in an organization should own software quality,” said Michelsen of the giveaway. “We want to ease the adoption process for SOAs by giving development and QA teams an easy-to-use tool to start testing Web services,” he noted. Calling such apps a critical SOA element, “usually the first toe in the water before a full SOA strategy is adopted and implemented,” he added that now companies have one less barrier to hurdle.

In K7.5, runtime and multicore rules are implemented, so the tool can now be used to locate potential performance issues.

need to make changes to the Ant XML file,” he said. An API now permits new or existing rules to be created or modified to cope with the needs of a particular program. “Since we’re not running code, when doing a static analysis it’s hard to know exactly how code will be traversed. This allows you to tune our rules to help identify false positives” or cases where the program isn’t flagging something that should have been flagged as a problem, Gordon said. “We’ve made it simple to take a specific validation routine and modify it to identify whether it constitutes a certain type of problem, so it will accurately identify whether the problem occurred or not.” In addition to the Defect and Security edition ($2,995), Klocwork also offers a $3,995-per-seat Enterprise Developer edition that includes the ability to use metrics to track project quality from build to build and the ability to graphically display errors and model the impact of change. Prices remain unchanged from the previous version, 7.1. www.stpmag.com •

ThinGenius Bostech’s Free ESB Covers Has the Mind The Middle Tier Of a Tester For companies deploying single-purpose applications such as those used by bank tellers or retail clerks, the thinclient device is a common way to save on PC hardware and maintenance. But how do you know if your app and its host server can handle the freight generated by hundreds or thousands of users? Load testing is the answer, but the tools tend to be pricey and require programming skills. Breaking that mold, says U.K. firm ThinGenius, is TLoad, its application load and monitoring tool for Citrix environments. “Load testing tools have been on the market for many years, but there are still a huge number of companies... experiencing performance problems and downtime, or seeing their systems failing because they can’t afford or can’t deal with the complexity of existing tools,” claimed ThinGenius COO Clovis Younger in a news release announcing TLoad 2.3, the latest version. TLoad requires no special skills, he added. It costs US$6450 for 100 virtual users. You install TLoad on a PC and execute your application and its use cases. The tool records keystrokes and mouse clicks, converting them to visual scripts and synchronization points for later playback as a virtual user. Scripts are stored and can be retrieved and reused in a tree-like view. Once a script has been stored and selected for playback, you select the number of users to be emulated and the load type. Tests can be stored and executed on multiple PCs on a LAN. Application performance, user counts, response times and server performance are captured and displayed graphically in real-time and can be saved in Excel-compatible formats. TLoad supports any application written for Citrix Presentation Server or MetaFrame. Check out the 30-day free trial at www.thingenius.com. Send product announcements to stpnews@bzmedia.com

• Software Test & Performance

Having trouble keeping Web integration testing under budget? If a free ESB would help, integration solutions provider Bostech might have an answer. Next month, the company is set to begin shipping ChainBuilder ESB, a Java Business Integration–compliant enterprise service bus that it will offer under GPL and commercial licenses. An alpha version is available now. ChainBuilder ESB delivers an Eclipsebased point-and-click interface that, according to the company, simplifies the assembly of Java components by displaying them in “a high-level graphical integration flow,” allowing companies to add their dissimilar back-end systems to a service-oriented architecture. By visually depicting an entire integration environment, developers and testers can more easily “consider the flow of their application integration and drill down on each specific component to define specifics.” Bostech CTO Eric Lu said that unlike other free or low-cost ESBs that center on XML translations, the JBI-based Chain Builder ESB focuses on XML messages,

which he said gives it the ability to integrate with non-XML back-end systems such as EDI X12, as well as fixed and variable formats. “Supporting these unique formats offers businesses a critical advantage to accelerate the incorporation of their disparate applications into an SOA environment,” Lu said in a statement announcing the alpha version. Available at press time was an alpha version (www.chainforge.net) released in late October. This nascent Web portal also hosts forums and other JBI and EAI resources. General availability is set for January 15, 2007. Bostech offers several commercial licensing options in addition to the common GPL, which requires developers to expose their source code. For ISVs and VARs that intend to include ChainBuilder ESB as part of their product but not license and distribute their source code, flexible pricing is offered based on the distribution model. Professional subscriptions start at US$995 and include remote training, software updates and alerts, legal indemnification and production-level technical support.

ChainBuilder ESB focuses on XML messages, which enable it to integrate with non-XML back-end systems as well as fixed and variable formats. DECEMBER 2006

Perforce

Fast Software Configuration Management

Introducing Time-lapse View, a productivity feature of Perforce SCM. Time-lapse View lets developers see every edit ever made to a file in a dynamic, annotated display. At long last, developers can quickly find answers to questions such as: ‘Who wrote this code, and when?’ and ‘What content got changed, and why?’ Time-lapse View features a graphical timeline that visually recreates the evolution of a file, change by change, in one fluid display. Color gradations mark the aging of file contents, and the display’s timeline can be configured to show changes by revision number, date, or Perforce Time-lapse View

changeset number. Time-lapse View is just one of the many productivity tools that come with the Perforce SCM System.

Download a free copy of Perforce, no questions asked, from www.perforce.com. Free technical support is available throughout your evaluation.

Pea k Perfor mance

The Balance Of Software, Ethics, Professionalism Counter to a common predeserve the respect afforded sumption, neither software to established, recognized testing nor any software professionals. Quite the condevelopment role, for that trary: Many members of the matter, is legally considered industry have earned— a profession in the majority through their education, of states in the U.S. expertise, accomplishments, Occupations related to softethics and community ware development are standing—the right to the becoming collectively recsame respect as those ognized as a profession belonging an established Scott Barber known as software engiprofession. The point here neering—and while some in the field is to distinguish between observing how a applaud this, others adamantly oppose it. person conducts him or herself individuWhat I find most fascinating, howevally and making assumptions about an er, is that few people currently in occuindividual based on a belief that he or she is a part of a recognized profession. Allow pations related to software developme to explain. ment—people who believe they’re part According to the State of Pennof a profession—are completely aware sylvania Associations Code (Title 15, of the implications of that belief. Chapter 1), which I’m referencing as an I’m not criticizing these individuals. example because it’s short, clear and I’m mentioning this simply because that generally representative, a profession: belief, coupled with a lack of awareness Includes the performance of any of its implications, is actually detrimentype of personal service to the public tal to the software development industhat requires as a condition precedent try as a whole—it can lead people withto the performance of the service the out significant knowledge of the softobtaining of a license or admission to ware industry to assume that some minpractice or other legal authorization imum degree of skill and knowledge is from the Supreme Court of required by law to carry a software Pennsylvania or a licensing board or development–related title (not to mencommission under the Bureau of tion creating uncomfortable situations Professional and Occupational Affairs for individuals who find themselves in the Department of State. caught unaware). Wikipedia discusses the concept of professions colloquially, as follows: A Pro Is as a Pro Does Before I go any further, I want to make In modern usage, professions tend to have certain qualities in common. A prosomething extremely clear. When I say fession is always held by a person, and it that software development–related occuis generally that person’s way of generatpations aren’t professions, I’m not implying income. Membership in the profesing that the individuals in these occupasion is usually restricted and regulated tions don’t conduct themselves in a proby a professional association. For examfessional manner. Nor am I implying that ple, lawyers regulate themselves through those who do act professionally don’t

• Software Test & Performance

a bar association and restrict membership through licensing and accreditation of law schools. Hence, professions also typically have a great deal of autonomy, setting rules and enforcing discipline themselves. Professions are also generally exclusive, which means that laymen are either legally prohibited from or lack the wherewithal to practice the profession. For example, people are generally prohibited by law from practicing medicine without a license, and would likely be unable to practice well without the acquired skills of a physician. Professions also require rigorous training and schooling beyond a basic college degree. Lastly, because entrance into professions is so competitive, their members typically have above-average mental skills. In both cases it’s clear that occupations related to software development fall short of meeting common criteria that qualify fields as professions. We certainly have no software development association that restricts membership, accredits training programs and enforces discipline on its membership. Another interesting piece of data that is not widely known: The United States, at least, has no legal precedent that establishes the tort of computer malpractice or of professional negligence in software development or software engineering. Needless to say, abundant legal precedent exists for the tort of malpractice and professional negligence in the recognized professions of law, medicine and engineering, for example. This makes sense, really. After all, you must demonstrate no minimum skill or knowledge to become a software developer or tester. In fact, you can become a software developer or tester with no training whatsoever. It certainly wouldn’t seem reasonable to hold an individual legally liable for negligence in having failed to apply practices that he wasn’t required, or often even expected, to know in the first place, would it? Scott Barber is the CTO at PerfTestPlus. His specialty is context-driven performance testing and analysis for distributed multiuser systems. Contact him at sbarber @perftestplus.com. DECEMBER 2006

Unfairly Accountable? What this means is that the company publishing the software (serving as the employer of software developers and testers) is legally responsible for ensuring that software is of appropriate quality—not the individuals the company employs to develop or test that software. This also makes sense, since an individual developer or tester rarely makes the decision to publish the software. On the contrary, in a recognized profession such as law, medicine, accounting or engineering, it’s not just the employing organization that is legally responsible for the quality of service delivered by its employees (though the organization may also be responsible); instead, the individual providing the service can be, and often is, held personally responsible for substandard services. For an example of how that accountability might work, imagine that someone purchased a software product that you had once contributed to (be it as a developer, tester, analyst or some other role). Now imagine that the purchaser found that the software didn’t perform according to reasonable expectations in some way. Further, imagine that after the call to customer support and the root cause analysis, it was determined that the bug/issue/defect was in your code, or in the area of the application that you tested or collected requirements for. Finally, imagine that the individual who purchased that software was then legally able to sue you, personally, for professional negligence or malpractice. This is exactly what happens when you become part of a recognized profession, and someone decides he’s displeased with the quality of your work. I don’t know about you, but the thought that I could be personally sued for not writing code to catch some obscure error condition or for not detecting some bug that appears only on some rare configuration that I didn’t get a chance to test based on the schedule and the company’s assessment of risk frightens me more than a little. Worse, what if I had detected and reported the issue, but the company had decided to release anyway? Worse DECEMBER 2006

still is the fact that, for all practical purposes, it’s not possible to completely test software—and even if it were, there’s no generally accepted standard that I could reference to demonstrate that I had done my job appropriately. Since we aren’t “professionals,” we don’t have anything to worry about. The worst thing that can happen to us for doing a really bad job of developing, testing or designing software is that we can get fired. With no legal precedent for computer malpractice or professional negligence, we can—and some do—get away with doing shoddy work with no real consequence.

‘Professional Quality’

•

until recently there was no nonprofit, professional-class organization dedicated specifically to software testing. Now there is. The nonprofit Association for Software Testing (www .associationforsoftwaretesting.org) is dedicated to advancing the understanding and practice of software testing. AST serves academics, students and software development practitioners. In September of 2006, AST adopted the ACM Code of Ethics as a series of principles to guide and govern practice among its membership. In the spirit of full disclosure, I’ll tell you that I am a founding member of this organization and the board that voted to adopt the code of ethics, but that’s not why I’m sharing this with you. I’m telling you this because I believe that many individuals who mistakenly refer to themselves as software development or software testing professionals do so in an attempt to express that they take their careers seriously, strive for continual improvement, are committed to quality software, and contribute (or at least try to contribute) to the advancement of their field. It is my opinion that membership in one of these organizations serves to make those claims about an individual and his or her work. As a member in good standing of AST, ACM or IEEE, you’re explicitly stating that you’ve personally accepted these responsibilities. In my view, that’s a powerful testament to your sense of personal responsibility for the quality and ethics of your work. And that matters more to me than whether or not I could be held legally liable. If you’re not already a member, I encourage you to look into ACM, AST and IEEE, and to give some serious consideration to joining one (or more) of these organizations that fits best with your area of specialization. ý

In a recognized profession, an individual can be held personally responsible for substandard services.

While I have no interest in being legally liable for the quality of software that I have no final say in releasing, nor am I advocating for a state license to practice software development or testing, the absence of state standards or legal liability is no excuse for doing less than excellent work. I do believe in personal responsibility and doing “professional-quality” work, as I’m sure that most software developers and testers do. So how do you demonstrate to your employers, clients0 and users that you, as an individual, are personally committed to doing our best work, staying up-to-date with your knowledge and training, and reporting issues and risks honestly and ethically? You can join an organization or association dedicated to your specialty that requires its members to accept and abide by a standard of ethical behavior. Several such organizations are relevant to software development–related occupations as a whole; for example, ACM (the Association for Computing Machinery) has a special-interest group for software engineering, and IEEE (the Institute of Electrical and Electronics Engineers) has a computing society, but

•

www.stpmag.com •

Mercury Remains The Perennial Favorite; Ant And JUnit Celebrate With Fresh Technologies BY EDWARD J. CORREIA AND LAURIE Oâ&#x20AC;&#x2122;CONNELL

he votes ar e in, and o nce again y ducer of te ou’ve selec st in g to ted Mercur o ls . We’ve eve that critics y as your fa n per forme of the obvio vorite prod a recoun usly less so countr y ha t p just to be su histicated e ve deemed re, a feat lectronic vo impossible Not only ting system . has Mercu s around the r y once a QuickTest gain achie Professiona ved the to l—but its tools have p spot—th LoadRunn earned eig is year fo e r, h T t e o st f D th ir r Other imp e remainin ector, and ressive finis g 20 first-p Q ualityCente hers were C lace spots. of top hono r ompuware rs in their a categories. were Apac Tools makin nd Pragmatic, each w he Ant, Ec ith a pair g the cut fo lipse TPTP for Softwar r the first ti , JUnit, M e Testers an me this yea icrosoft’s V d Mindree with Purif isual Studio r f’s SOAPsc yPlus. T e am Edition ope. IBM R So sit bac ational rep k, relax a eated this y n d deemed the ear read abou best in the t the tools th industr y. T a t you—the hese are th readers—h e tools nam ave ed Tester’s C h o ic e . GRAND

Grand Prize

PRIZE WINN ER

Enhanced Mercur y Q team colla uickTest Pr boration c manager apabilities— and the a ofessional b in il cluding a it y to share surely playe new objec function li d a role in t reposito braries ac thrusting the stack th ry ross tester Mercury’s is year. workgrou Q u ic k Test Profe ps— Also recen ssional to tly added ing: new the top of to the auto keyword m drag-and-d m a ti o n a to n rop test-ste agement o l fo r fu nctional a and enha p construct rate debu nd regress ncements ion, XML gger that ion testto keyword output for identifies test-result errors whil h a ndling su reporting e building ch as a and a new or mainta and more ining test DATA accucases.

/TES LOAD/PERFT PERFORMANCE and ORMANCE T Mercur y L EST oa

junction w dRunner ith a load Load testin generator es of users to simulate g might h . T he Analysis a massve saved sy talk show module gra o f these load ndicated ra host Sean phs the resu scenarios. dio Hannity so ment in O lts me emba ctober; his rrass“You’re a G car giveaw F U N re CTION at Americ ay Web ap an” plication shortly aft TEST AUTOAML TEST and ground to er it went a ATION live due to halt In this Mercur y Q high user year’s co uickTest Pro volume. mpetition LoadRunn fessional Mercury , Mercu er scored a QuickTest ry’s double vict top spot in P ro th fessional is year in b ory, nabbin both the d got the n oth our fu g the ata perform performan od nctional te automatio ance and ce categori sting and n categori lo a e d s. te e st s. LoadRun it R s e e a ase of use ders rated ner consi and impre it #1 for sts of the Generator ss re iv c e o fu rd a n d p la Virtual U (VuGen), nctionality Controller y back UI ser : Users ules. VuGe scripts, en and Analys in te ra c ti o n creates te a b is li n n mods g a in st scripts th s te st p back and u t customiza meterizati at can be p modified tion and p on. layed for differe aradata corr nt test para With supp elation a meters, ort spann nd error Controlle in in g cluding .N major envi handling. r execute ET, Web se ronments The s scripts a rvices, Java mainfram nd works e , ERP/CR and Wind in conM, ows, Quic word-drive kTest Pro n approach ’s keydigs into u nderlying object

www.stpmag.com •

g and criptin tions d e t a c of fun integr rough he pain What’s not t h t e s s a e i e rt o n. prope tures t o automatio t ing fea g g g in t u s b de adju g and al testin nd ? to like MENT a

e a choic ffering as new o , s ie ll ilit capab g as we tegriorting ted updatin in p e d r n a w time n and ne l or automa u r , ua -time of man its compile r o f rules lyzers. ty ana ST/

GE E MANAENT U S S I nter / DEFECTQA MANAGEM for Quality Ce in the testTEST/ y TestDirectoranagement lionuble-wham-

E OBILE T M / D E D EMBEDRMANCE rofessional obile and m M d O P e F t d s R e d E T e P IB k b y Quic for em placed

r Mercu ster’s Choice this year dis st year and e on la th Te rcur y st

hw As kTe , Me e, whic . Quic ations ility applic s Test RealTim ose second r its ab h l’ cl o f a a s n r io e n t i wit e Ra read r teams ar cam ed among g e Mercu the project m ed its third do t managey n i t s s i e th ution dt shin g or tes ent an ing the exec ter, sional s Tamin , Mercury sc Web-based n m e e f p C o o r l y t e P a s ev na alit vide d ols for autom esting. with it ing are for Qu issue manst to pro t o e year ector t / r h n t i t e c or mo D io f e iv t s f t o s s my Prize f ur de egre n, Te abora ories. r o a d o ll g s i n d h e t o t t a t r n c r u o a l a o b c so nt nal s in ar’s G supp ment ageme pan of g honor functio rded this ye rofessional s n n f p i a e o o d t m id u g l w A awa st P , inc th to a i s e t o s w nabbin and test/Q T e n ls k r s e A t c u i s o t m Qu on maj r nt s fea analy verall, target envir r all ngio ageme irector offer g business o e s f e d t s n o a v of Visual app tin ers TestD Web T 2.0, variety s, assis , test manag A with test E r e L e g .N b u M , h m X e veX al plug ,Q ion L and 2 and Acti team m ents definit option prior eration rs with bug M p h T o g u H d 3 o n r in a s em pe ers, W 1.4. Th ET version requir design develo modbrows N a SWT ith test porting, and mploy core . er, and v w t a h s J s g r e i d t e — h n t n e e a d ne n r a e n c n ic d a s a m n Ba 1.1 iebel age also ion a sers c SAP, S s, JDK ct man Test Lab e tool , operat tDirector u m t je h f r t o o r o , s S F p in ws ple n, s of Tes Windo Oracle, Peo aspect ent, Test Pla ndalone fixes. o 2.0, ing all a t t r r m s e s. o e v s f g a m o a e c e r an ad he onsyst ules ents M ement—eit l envir apps m enterprise a m e b o ir l u g g a M Req Mana within and IB NT efects rated g e t and D AGEME n N i A r o M hope s n D UIL B he. We are c / solutio a M p A C , S t softw Ant ’s circle ment. ST E pache o the winner rs of the An t system T A S E C se VI en io vicme t stay. U agem ble eb ser Welco EB SER er Stud y your n and man e tool availa o j n SOA/Wware DevPartnld of SOA and tWner Studio e k u o i i o -l t y e a k autom the best ma Compu rave new wor are’s DevPar build build b w it’s t u e a p h d h t l res its t m i o o u d t In s e b C t e , n r n r ag from va. w, A u ca s yea rence ly kno a in Ja s e ion, yo rvices for v es, thi preme. f t u i a f i J d io d r e v o r l f apps. se na s ob su ajo s. on-Java erreader ofessio ts and Web L, a m reigns m n r s r r P M o A o f f e t X l h la n in y to p d too With t , componen Windows p ? Tr y rmatio minant buil h Ant’s abilit e Ant n o s f o e i n n t iv i o t u i o iv it at na sol a pred long w applic t .NET and tions, g hich addinsive make, l file, a quired func prehe es two nd ,w m sof d m e o .x o k lu r d c a c ic il e M an m wn r is bu ich in more h nt a o h t h e s a s. T y w it m it , d f e il n e mand Ne itio nag d; any o ortab m com rise ed change-ma Recor e orm m ibility and p k p t f r s c y e a t s r n T ic the flex the E ts and -specif lution more tools: ing so requiremen out latform p n o tional roject-track s h s er ug relie /p h deliv rs thro MANCEm Edition defect R e, whic to develope O il F c R n E o ec es o Tea ST/P and R chang .NET TEoft Visual Studi Team ment e . r e i l u c y q tudio c s S re o e l f r a i c l i u Web M Vis er oject re Test s again with ormance and ges. a the pr w r t f e o k S ua erf hec ore T lang soft sc er, its p TEST rtner Security Crity Checker Y nd .NE Micro oftware Test T a I osoft’s R r io U ic d a , u SEC ware DevP rtner Sec y categor y al Stu ith M ion S u w it is ile d d V E e t r a fo gra nts ag rit Compu are’s DevP g suite is inte pleme e secu curity analyin r t m h e s i t t e s t d e n w i n T u a se re Comp top honors e app Server velopSoftwa ation hensiv d ies de d e f e i r n l b p u p h b o m g a m F n si rou e co Team rity th ing th victory. V2.5 ules n secu repeat r o 5 i y t 0 it a 0 il c 2 i pl rab l’s sis too ASP.NET ap anded vulne r p o x f e t , s n me pdate ated u autom

• Software Test & Performance

DECEMBER 2006

aturity Model e Capability M th d an es ive gi lo methodo ensure effect MI) to help M (C n io at Integr ement. process manag

IS IC CODE ANALYS M A N Y /D C TI TA S urifyPlus P IBM Rational

Rational was ht year, IBM ig ra st nd co dynamic code For the se for static and ol to ost be e voted th s and other pr memory leak ng di d fin an r ce fo is analys rforman d conducts pe an s, ct fe de gram ysis. d ux, Unix an coverage anal rately for Lin on iti ed se ri Available sepa rp an ente ications or in tomatWindows appl ains a set of au nt co us Pl fy ri Pu e, d .NET lanfor all thre C/C++, Java an r fo s ol . to e d Visual Basic ed runtim isual C/C++ an V as l el w as guages

find relief ables, you may er liv de d an s ALM ments, defect e Commercial ’s winner in th ased -b eb W ’s ic from this year r, Pragmat ne an Pl l e ar w ep lp you ke al class. Soft ent tool, will he em g ag in an ud m cl e life-cycl am tools, in e air with its te synthe balls in th d MS Outlook an ch folders ar se disal d ic an ch g ar in hier ment shar as well as docu n, tio za ni ro ch ement. lendar manag cussion and ca

ST/QA COMMERCIAL TE 0/SEAT LESS THAN $50 er

ect Track assurance capa Pragmatic Def ing and quality

al test r seat, For commerci an US$500 pe costing less th ol to from a r in s ke ie bilit ect Trac s chose Def er ad re e or m ftware. Pr ag m at ic Pragmatic So to n ot e th at g in st re te It ’s in gory and was ot in this cate sp p to e th te go ri es of achieved up in th e ca er n n . d ru n a a E C en RMAN Issue Tracking n ot ev ent or Defect/ em ag JAVA TEST/PERFO ng an ui M ig A tr Test/Q some in nly do make FREE TEST readers certai ur O ds awar es. JUnit ster’s Choice Tester’s Choic mer to the Te Java r fo k or ew Another newco g fram ee unit-testin e Java is JUnit, the fr arks in its nativ PERFORMANCE m p EE to FR ed iv ce of re st t ni be apps. JU so was voted Tools Platform Eclipse TPTP tegory, and al Performance d an t st unit-testing ca Te e. bl se la The Eclip ce managemen A tools avai publicized free performan t st bu the free Test/Q at be t e th us e th ug is gu A t ight ar projec leased in pletely u all. Some m JUnit 4 was re ntained a com ailable, said yo av co ol It true even with . to at be th ld va to ou atement w st features in Ja widely prior g w in ne ed r fo ec t pr or the and supp ll received. free. rewritten API n tracers re obviously we we s out the word ie lit bi pa Java applicatio ca es ud cl in t 5. The new ec ance inforThe proj yzing perform al an r fo rs applications, and profile ST SUITE d distributed an l ca INTEGRATED TEy Center lo r fo test authoring mation onitoring, and m p ap Mercury Qualit ite of 2006 isn’t exactly com nit. ed at ic sophist grates with JU g su at the ubiqols. It also inte th to The top testin n se io ri ut rp ec su ex and field. It’s no g two douing out of left tion to scorin di ad in — ’s ry ar ye uitous Mercu ANY Prize in this d the Grand M A NEW COMP O FR park e N O th TI of t LU ble-headers an O ou S ll BEST knocks the ba Pscope with competition— ter. en C y Mindreef SOA lit scene in 2002 ua Q ed as -b ud cl eb in W came onto the f ts ee the en dr of nm in with its d M ro ar vi n he Whe d even ort for all en g SOAP, few ha heavy hitin an Offering supp is st m te th ss r ro P, fo G SA k ol d to a naries Fran T, Oracle an d sting tool visio mated softan te to E ing J2EE, .NE au ow IC N s: ft . ar So st rm g te ercury’s bi se top-selling r, M to ho es w ec ir ud n, D cl st ku by in os Te r te quired ercury and Jim M ultimately ac ms such as M al, Mercury hecker were ware testing ge C on ds si arts. n es ch ou of e B Pr th g ckTest ing ain toppin st ag Te e s ar es e, cope oc ar Mercury Qui w Pr iness Compu n 5.0, SOAPs Mercury Bus new at versio ly rd comha e WinRunner, le th hi W strips away Service Test. king tool that ea pers br lo and Mercury nd ve ou de gr is a ges to help e SOAP messa th Web of of es se iti au ex pl the root-c LM ickly identify .ý qu s on COMMERCIAL A00/SEAT er iti st ed te d am an ’s also a te 5 re $ he N T A s. m TH le S S LE services prob arePlanner Pragmatic Softw g too many test cases, requirejugglin If your team’s

DECEMBER 2006

www.stpmag.com •

THE RUNNERS-UP 2006 DATA/TEST PERFORMANCE

EMBEDDED/MOBILE TEST/PERFORMANCE

Compuware Vantage Embarcadero Performance Center

IBM Rational Test RealTime Coverity

FUNCTIONAL TEST

SCM/BUILD MANAGEMENT

Empirix e-Test Suite IBM Rational Functional Tester

STATIC/DYNAMIC CODE ANALYSIS Eclipse Test & Performance Tools Platform Parasoft Jtest

TEST/QA MANAGEMENT Borland SilkCentral Test Manager Compuware QADirector

DEFECT/ISSUE MANAGEMENT Microsoft Visual Studio Team Edition for Software Testers Mozilla Organization's Bugzilla

LOAD/PERFORMANCE TEST

IBM Rational BuildForge Catalyst Openmake

.NET TEST/PERFORMANCE Mercury LoadRunner NUnit

JAVA TEST/PERFORMANCE Mercury LoadRunner Eclipse Test & Performance Tools Platform

INTEGRATED TEST SUITE Microsoft Visual Studio Team Edition for Software Testers Compuware QACenter Enterprise Edition

COMMERCIAL ALM UNDER $500/SEAT Rally Software's Rally

Microsoft Visual Studio Team Edition for Software Testers IBM Rational Performance Tester

COMMERCIAL TEST/QA UNDER $500/SEAT

SOA/WEB SERVICES TEST

FREE TEST

Parasoft SOAtest Empirix e-Test Suite

Eclipse Test & Performance Tools Platform Mozilla Organization's Bugzilla

SECURITY TEST

FREE PERFORMANCE

Empirix e-Test Suite Watchfire AppScan

JUnit Mozilla Organization's Bugzilla

TEST AUTOMATION

BEST SOLUTION FROM A NEW COMPANY

Mercury WinRunner Borland SilkCentral Test Manager

Solstice Integra Suite Agitar Agitator & Management Dashboard

â&#x20AC;˘ Software Test & Performance

Intel VTune Performance Analyzer

DECEMBER 2006

Sizing Up Automation Candidates Selecting Which Tests, When To Automate Them, and Which To Take Off the Ticket Entirely By Bob Galen

o there you are. You’ve done a little research on test automation, made the business case to upper management—and they bit on

the proposal. To your surprise, they supported you all the way and are extremely excited about how much faster testing can really move. Or… Upper management comes to you with an edict to start automating your testing: You’ve got to improve testing cycle and turnaround time—and do it with fewer resources. They clearly believe automation is the only way to achieve these goals. You’re given a nearly blank check to quickly bring automation into play in your next major release—three months away. In either case, you’re in an enviable position, poised to begin developing test automation. If you read part one of this Bob Galen is the author of “Software Endgames” (Dorset House, 2004) and the founder of RGCG, a technical consulting company focusing on agility and pragmatism. He can be reached at bob@rgalen.com.

series, you’ve learned of the tremendous boost automation can provide your team members in terms of its technical challenge and their contribution, at the same time wowing the business folks by driving testing cycle times down and coverage up. Frankly, though, this can also be an intimidating time—especially if this is your first time trying to sort out where and how to begin, which is exactly the focus of this article. One of the basic assumptions I’m making here is that previously, you’ve been creating and running test cases manually as an organization for a while. That is, you’ve built up some sort of repository of manual test cases that are potential automation candidates. Given that you can’t automate everything at once, the key question—and challenge— becomes where to start and how to properly orchestrate your efforts over time. I’m also assuming that you have limited resources and little time to produce visible results. Because if yours is like most organizations, you have responsibilities other than automation, such as testing and releasing your products. This makes prioritization and establishing a work balance critical. www.stpmag.com •

TEST BALLOTING

ANTI-PATTERNS: THINGS TO AVOID WHEN STARTING AUTOMATION Anti-patterns have been applied to software design, coding, configuration management and just about any activity within software projects. Exposing what not to do will help set the stage for some of my automation recommendations, so here are a few general anti-patterns for selecting good test candidates for automation development. We don’t need no stinkin’ criteria. It amazes me how many groups simply dive into automation development without a strategy to select which test cases to automate. They simply start automating somewhere within their pool of test cases, often picking an arbitrary starting point such as the first character in the test’s name and then moving serially forward from that point. Another mistake of the “No Criteria” anti-pattern is not to reevaluate your lack of criteria as you make progress. Good start, then a stumble. I also see teams that start well; for example, by picking a set of “low-hanging fruit”—automation candidates that make sense to initiate the automation program. Typically, the set is small, intended to get the effort going quickly and to realize some short-term successes. However, after the team members accomplish their initial plans, they fall back into a “select anything you want” mentality toward automation. Don’t ‘just do it.’ Another frequent antipattern is the view that all test cases need to be automated. This creates excess churn because the team is frenetically trying to do it all—often working in parallel with a mainline software project while struggling to automate test cases on a moving functional target. This pattern also drives the mentality that all test cases need to be automated independent of the level of effort associated with that endeavor. This is simply not business realistic. In tools don’t trust. This anti-pattern is focused on the myriad automation tools available. Often this abundance can lull a team into the illusion that tools will take care of the details surrounding automation—thus removing their need to understand the tools and technologies being used. It also prevents teams from understanding the strengths and weaknesses of each tool as it relates to different technologies. And trust me, every tool has problems that need to be considered. When not to stay the course. In this pattern, teams do everything well at the beginning: They take an inventory of their test cases, put together a thoughtful ordering and start building good automation that contributes positively to their projects. However, teams sometimes become stuck in their initial list and never adjust it for discoveries and changes as they progress. As time moves on, the original approach can become irrelevant.

• Software Test & Performance

Selection Criteria – Patterns Anti-patterns are an invaluable aid in learning what not to do. Avoiding the pitfalls of anti-patterns described in the sidebar, here are some general selection patterns that normally govern my thinking when selecting test cases. Consider these practices when establishing your own automation efforts. Pick low-hanging fruit. As I described in the anti-patterns section, selecting low-hanging fruit, or the tests that obviously make sense to automate, is a good way to generate initial momentum. These are not necessarily the most impact-producing automation targets, but are usually small, localized, well-understood and stable functional components that are generally quick to implement. I’ll usually focus new automation developers on these cases for a lower learning curve and to measure their capabilities before assigning them more challenging work. A final important part of the pattern is making your successes visible—both internally within your testing team and across your organization. Showing off early success is an important part of the pattern. Flesh out the infrastructure. All good automation efforts establish an infrastructural layer that wraps their tools for efficient automation development. Usually this starts with implementing automation development templates, standards, naming conventions and other guidelines that enforce consistent practices and procedures for development. Often it also involves writing some wrapper code that programmatically reinforces the consistency and serves to reduce startup time and learning curve. These practices need to be verified and debugged, so selecting suitable test candidates to flesh out and verify the infrastructure becomes important. Usually these candidates are distributed along the architectural boundaries of the application under test

•

(AUT) so that the infrastructure is verified to support all aspects of the application environment. Minimize rework. Two factors can drastically affect your efforts and influence rework: requirement stability and application stability. While you can’t always control either, you can choose when to automate, and thus mitigate their impact. In my experience, the easier factor to control is requirement stability. In this case, you simply wait until you have sign-off or agreement on requirements. Another indicator of requirement stability occurs when the development team begins construction; normally, this is a good indication that it’s safe to start the automation. Application stability is tougher to control. I usually like to trigger automation development after we’ve at least had a chance to exercise features manually. If manual execution is impossible (for example, with an API), any demonstrated testing with positive results can serve as a trigger for automation development. A good strategy for removing both of these variables is to defer your automation construction until after the specific application release is deployed. Complexity and packaging tempo. One of the more important factors to consider is difficulty of implementation. I wish more product development teams would consider this when developing their products. If you have 100 test cases that you want to automate and they’re all exceedingly complex, the deck is stacked against you. When developing an automation release package, I prefer to create a more normally distributed set of test cases—say, 25 percent very complex, 50 percent moderately complex and 25 percent relatively straightforward. I’ll also create a goal for the package that represents one of the overall patterns in this section. For example, after development of my first Low-Hanging Fruit automation

The best choices for automation are tests that are quick to implement and that will have the greatest impact on your project.

•

DECEMBER 2006

TEST BALLOTING

package, I’ll often go for a release centered around cycle-time or speed, as explained in the next section, Project Impact: Speed. This way I immediately start leveraging automation impact on the project. Regardless of the approach, you should develop a packaging strategy that uses goals and short iterations to create a tempo for your automation release. It’s also a good idea to avoid underestimating considerations of project hand-offs, as “Consider Hand-offs to Avoid ‘Hail Marys’” illustrates. Project Impact: Speed. As testers, we always want to maximize the impact we have on our projects. Automation can drastically impact testing speed, so examining how quickly an automation package can be created and executed is a compelling selection criteria. It also pays to select for automation test cases that take the longest time to execute manually, because these too can have a drastic effect on testing cycle time. Of course you need to consider all aspects of time when picking these candidates, including test setup, data preparation, execution and results analysis. Clearly you don’t want to automate the smallest or fastest tests first if you want to make a high impact on cycle times within your project.

Project Impact: Risk. One of automation’s most powerful uses is risk mitigation. From that perspective, you may want to choose candidates that align with some of the more challenging areas of the application—areas where you know the development team will struggle and where you can make a difference. This may mean that you take on less so that you can make a more powerful impact on a product’s maturity and stabilization. One key practice to consider is the Pareto Principle, also known as the 80:20 rule. As applied to software development, this means that 80 percent of the bugs (or risk) will be created by 20 percent of the application. If you can identify and isolate that 20 percent and apply your automation efforts toward it, you’ll be well on your way to effectively mitigating project risk. Project Impact: Maintenance. One of test automation’s hidden properties is that it is indeed a software development project. As such, it too succumbs to maintenance issues—in both the short term, as you try and automate to a changing application, and the long term, as the application evolves and matures. I’ve found that as much as 30 percent of my automation development time is spent in handling tradi-

TABLE 1: THE UPS AND DOWNS OF METHODOLOGIES Methodology Waterfall

Automation Selection & Construction Implications While the waterfall method has been challenged as non-agile and inefficient, its need to define requirements early offers benefits to automation, including the ability to select candidates earlier in the SDLC. Waterfall also makes selection more global in nature, thus permitting more multifaceted decision-making. One down side is the frequent need to hold automation until a subsequent release before being able to positively impact the product with automation.

RUP – Rational Unified Process

Generally, RUP is front-loaded with analysis and iterates through construction.This means that you should be able to gain a firm handle on automation candidates roughly onethird of the way through the SDLC. However, implementation will be iterative in nature. It’s best to connect the automation strategy to the development construction strategy, with a focus on early construction, iteration speed, risk or coverage goals.

Agile methods

Agile methods disrupt traditional automation methods and approaches. The first disruption point is the focus on upfront automated acceptance testing.Typically, the tools used for this fall outside of traditional test automation tools and preclude any sort of reasonable reuse for other sorts of automation. The finely grained iterations make it difficult to focus on more than one specific goal or criteria per automation release. It helps to mirror automation releases with their development counterparts, even though they might be skewed by an iteration.

DECEMBER 2006

tional maintenance-level activities. And from my point of view, this was using well-architected automation solutions. So the potential for worse outcomes is great. One way that I handle maintenance work is to steal a technique from the development life cycle: scheduling focused maintenance releases for automation. I’ll accrue maintenance work for these sorts of releases and time them to coincide with possible downtime within my projects. This also allows me to better understand, measure and communicate the costs. Clearly, you don’t need to consider all of these patterns in every selection decision, and they all have different weights depending on your specific context (see Table 2). What is important is to create a breadth to your analysis. Focus test case selection on the ability to make a visible and fundamental impact on the project. Automation efforts should be seen as a clear differentiator when the team discusses project success factors.

Selection Criteria: Change Is Good One of the challenges associated with selection criteria is that they don’t remain constant, or at least they shouldn’t. All sorts of conditions change with your automation projects that should cause you to pause, reflect and adjust your criteria. To illustrate my point, let’s examine some of the primary conditions that usually cause me to stop, think and change. • Changes in skillset, either positive or negative, can certainly change your criteria. Some of the factors that can influence this include attrition, outsourcing, changing project priorities and changing tool sets. Closely moniwww.stpmag.com •

TEST BALLOTING

tor your performance capabilities and adjust as necessary. If your capabilities or those of your team increase, you can and should take on more with each project and automation iteration. If they decrease, you must take on less, which means placing more importance on the selections you make. • Changes in application technologies are often adopted by pesky developers who sometimes disrupt ongoing automation efforts. If you’re lucky, you get an early warning so you can experiment to ascertain the impact prior to product release. However, most of the time, you’ll become aware of automation implications at the point of implementation. If I anticipate technology changes that will impact automation tools or strategies, I often plan for a small impact evaluation iteration. Within it I look to fully understand the impact of the changes, evaluate the impact to my tools and infrastructure, and carefully estimate the change impact to my existing set of automation. Once I have a good feel for the full impact, I’ll adjust my plans and criteria as required. • Changes in team balance occur as people enter or move around within an organization. When planning my testing and automation efforts, I often think in terms of producers (developers) and consumers (testers). This ratios makes developer-to-tester important for capacity and workflow planning. If you get a drastic change

in your ratio, it will certainly have an effect on your automation capacity. So reevaluate prioritization to ensure that you’ll be producing the more relevant candidates for your rate of consumption. If you don’t have a dedicated automation team, this can become an ongoing problem as you reallocate automation resources for other proj-

ect-driven testing activities. Multitasking resources can also create this effect. • Ongoing setup time, or the time it takes to set up and tear down your automation runtime environment, can be a significant factor in how you select automation candidates. This becomes particularly important in some database applications that require vast amounts of time-sensitive data to physically set up and run the automation. As a risk mitigation strategy, I prefer to attack time-sensitive test candidates quite early in my automation

CONSIDER HAND-OFFS TO AVOID ‘HAIL MARYS’ Proper hand-off of your automated tests is as important as the development itself. If the people executing the tests aren’t properly trained, all your hard work could be for nothing. I once worked on an automation effort that practiced some of the steps detailed here. We did a solid job of considering how many tests to automate, which ones to work on first and how to deliver the automation code. I was quite proud of the effort. However, after a few development cycles, I was surprised to learn that only about half of the automations would run in the regression cycle. After some digging, I discovered that the root problem was a poor hand-off from the automation team to the execution team. In too many cases, the team running the tests was having problems with setup, physical execution or figuring out how to analyze test failures. It was then that I realized how important deployment planning was to any automation effort. Not only should this cover training and the physical hand-off, but it should also include usability as a design goal. Clearly, automation that sits on a shelf is a benefit to no one.

• Software Test & Performance

efforts. Often, I don’t understand the full implications of the setup nor the impact it will have on my scheduling of other, much shorter, automation. • Evolution of maintenance costs can be severely impacted by automation, depending on how it’s architected. Part of this burden is driven by attempts to implement too early— before interfaces or functionality within the application have stabilized. Another aspect is internally driven; for example, when you make infrastructural changes or upgrade versions of your toolset. One reality of maintenance costs is that you must pay immediate attention to them and address them early. Fortunately, such costs are frequently cyclical in nature, and once addressed, allow you to fall back to your normal selection scheme. • Evolution of development methodology will most likely impact your automation efforts (see Table 1). The most drastic example of this can be found in organizations using agile methodologies. In these cases, you’ll want to shift your automation focus toward individual development iterations, while helping to automate the unit and acceptance testing activity within the context of each. In addition, it’s a good idea to automate candidates as they exit the iteration. However, due to the potential for tremendous volatility across the feature and interface set, it’s best to carefully consider the stability of each feature or candidate before implementation. • Changing business conditions come primarily in two forms: budget and investment or business value and requirements. These changes often have the same effect as changing skillsets or technology in that they cause you to change your automation selections, either positively or negatively. • Retirement consideration becomes important as tests become dated. Retirement is the removal of an automated test case from the execution DECEMBER 2006

TEST BALLOTING

TABLE 2: CANDIDATES WEIGH IN ON THE ISSUES Selection Factor Project Impact

Considerations Will automating this test directly help your project’s speed, coverage, risk, and/or development nimbleness?

Importance: New / Mature app

Complexity

Is the automation effort simple to implement, including data and other environmental challenges?

H/H

Level of Effort

How much time is required to implement the automation, including time spent on packaging with other work?

L/H

Early Requirement vs. Code Stability

Will the application requirements or early coding prove stable within the scope of the driving project?

L/M

Ongoing Maintenance

How volatile will this code be in the longer term, will the functionality change/evolve?

L/M

Coverage

Will automating this test cover critical application features and functions?

H/M

Resources

Does your team have the human, physical and data resources required to run this test?

H/H

Hand-off Efforts

Does the test execution team have sufficient skill and time to run this test automation?

L/H

L = Low M = Medium H = High Once you’ve started the automation effort and are comfortable with your techniques, you should examine test selection from a wider lens. This prioritization step takes time and experience to achieve, so consider these factors before selecting candidates. Automation decisions also can vary depending on the stage of the application’s life cycle. It’s helpful to divide selection criteria into three primary categories: 1) Automation for start-up efforts; 2) Automation for ongoing efforts; and 3) Automation for mature or end-of-life efforts. The checklist also might evolve as your automation efforts mature. For example, I’ve assigned sample weightings for new versus mature applications of Low, Moderate and High for consideration. For new application automation, the Complexity, Coverage and Resource areas are the ones that should be driving the priority of your automation decisions. Once you focus on what’s most important, you can rank potential candidates from 1-5 in each area and arithmetically select your candidates.

environment or a reduction in the frequency of execution. Just as software incurs technical debt over time, so too do automation efforts: Test cases can lose their relevancy and connection to the current application. What once was a high-priority candidate is now something that needs to be executed infrequently or not at all. As part of my changing selectioncriteria planning, I usually drive reevaluation of automation from a retirement perspective. You should formally note these changes and, if removing tests, store them in a retired automation archive. Select a regular interval for reevaluating automation environments— either on a quarterly basis or before starting a significant new project. That way, you evaluate the landscape cyclically and consider any important changes that might influence an adjustDECEMBER 2006

ment of the overall selection criteria.

Shifting Strategies for Selection These tips should help you plan and build a strategy for selecting automation candidates. And don’t be intimidated by the number of selection criteria included here—I’ve found that when I develop selection criteria that map to my overall automation strategy, I actually develop more automation, have more impact on the business and projects, and increase the visibility of my testing team. One final point: Don’t ever think that your automation efforts are done, because test automation is an ongoing challenge. Developers need to perform maintenance and stay on top of new tools, techniques and products. Next month, I’ll examine ways to build a solid business case for automation development. ý www.stpmag.com •

â&#x20AC;˘ Software Test & Performance

Testing Early, Often And Wisely Can Lead To Solid Results

By Moty Aharonovitz

s QA and testing professionals, our primary job is to find defects in software builds. Fortunately, however, most of us have moved beyond

exposing and tracking bugs to the more critical role of ensuring that our company’s software meets customer expectations before it’s released. To do this, many organizations have recently embraced requirements-based testing. Teams can be more effective using RBT because it enables them to target the root causes of quality issues by being tightly coupled with application requirements and integrated through various activities throughout the software life cycle. As a result, RBT helps organizations deliver measurably high, systematic and cost-effective test coverage, ensuring that what teams test is what matters to their customers. The challenge as RBT rolls out to our organizations is to couple its processes with existing QA and testing best practices. To that end, I offer three practical suggestions: 1. Test early and often, so that testing becomes a parallel activity to the development process, a constant pursuit that spans all roles and makes all stakeholders aware of Moty Aharonovitz, senior director of Product Strategy at Borland Software, has more than 15 years’ experience in software development. He is responsible for solidifying Borland’s Software Delivery Optimization vision and support of its Lifecycle Quality Management solution. He can be reached at moty.aharonovitz@borland.com.

www.stpmag.com •

RBT SUCCESS

ments defects are the result of poorly written, ambiguous, unclear and incorrect requirements. The other 50 percent of requirements defects can be attributed to incompleteness of specification (requirements that were simply omitted.) We can conclude that the two primary issues with requirements quality are that requirements and specifications are incomplete (due to lack of input from users and other key stakeholders), and that requirements are specified poorly (by domain experts who lack the skills to define consistent, accurate and testable requirements, and whose tools do not allow for efficient validation and/or review of the requirements with the users).

FIG. 1: DISTRIBUTION OF DEFECTS

7% 10%

56%

27%

Requirements

Design

Code

Other

Problematic Test Coverage quality objectives. 2. Test with your head, not your gut to inject repeatability and method to the test planning process to make test coverage more predictable. 3. Test with measurement and improvement in mind to quantify the status of deliverables and activities so that management can oversee quality initiatives across the IT application portfolio.

The Quality Crisis The Standish Group’s Chaos Report and other studies describe what has been the industry reality for decades: The majority of software projects fail to achieve schedule and budget goals. Poor software quality is a primary factor behind many failures and often results in massive rework of application scope, design and code. Such rework extends release cycles and consumes significant additional budget. To reduce software failures, you must better understand the quality initiatives behind the products being developed for today’s global economy. Through personal experience, I can tell you that the two primary reasons behind poor application quality are defects in requirement specifications and problematic system test coverage.

were wrong from the very beginning. Requirements of complex software applications are often negotiated through two concurrent dialogues that continuously evolve throughout the project life cycle. These dialogues focus on two questions: “What do we need to build?” and “What can we build?” The quality of these dialogues often determines the ultimate quality of the constructed application. A dated but still relevant study by James Martin (“An Information Systems Manifesto,” Prentice Hall, 1984) shows that the root causes of 56 percent of all defects identified in software projects are introduced in the requirements phase (see Figure 1). Further, the study showed that approximately 50 percent of require-

When quality is an afterthought, quality verification efforts typically begin only once code has been completed. At this point, the testing team is under the gun to certify the application as quickly as possible, and such certification is often perceived as a bottleneck that prevents deployment into production. In this environment, it’s not only difficult to ensure that the requirements are correct, but also to properly plan tests, to ensure correctness, completeness and solid coverage of requirements, and to gain visibility into the various quality aspects of the tested application. No wonder this often proves to be a costly and frustrating exercise to all involved stakeholders. On top of that, achieving satisfactory

FIG. 2: RBT PROCESS FLOW Logical Test Case Design

Requirements Quality Validate Against Business Objectives

Map Against Use Cases

Ambiguity Analysis

Domain Expert Reviews

Structure/Formalize Requirements

Validate Requirements

Define/Optimize Test Cases

Fix Requirements Validated Test Cases

Test Cases Quality Review Test Cases By Reqs Authors

Review Test Cases By Domain Experts

Logical Test Cases

Defects in Requirements Specifications Some of us aren’t surprised when we hear about end-user complaints and lack of adoption for applications that went through a rigorous QA and testing process that found few bugs. The reason we aren’t surprised about the uproar is simple: We know that the requirements

• Software Test & Performance

Test Execution Design & Code Quality Code Review Test Cases By Developers

Review Code with Test Cases

Complete Test Cases

Review Design with Test Cases Execute Tests

DECEMBER 2006

RBT SUCCESS

test coverage is challenging. The complexity of modern applications—in particular, distributed application—makes it more difficult to cover all of the possible scenarios due to the sheer number of alternative paths through the application. Any attempt to systematically (or automatically) consider all possible combinations results in a set of test cases that is too large to manage and takes a very long time to execute, which can mean further delays in certification. In addition, change management has become unwieldy in many organizations, as application requirements change hourly or daily. However, without the ability to properly manage these changes, traceability between requirements and test cases is often lost, making it even harder to ensure continuous test coverage.

RBT Best Practices: The Big Three Requirements-based testing is an approach that is implemented through well-defined processes that target the root causes of quality problems mentioned previously. Even if your organization is not using the practice today, you’re probably familiar with the key processes within RBT, highlighted in Figure 2. In addition to existing testing processes, organizations should adopt the “Big Three” best practices noted above to be successful with RBT: 1. Test early and often. As soon as requirements, design and code artifacts are ready, review them against business objectives, requirements, use cases and test cases. Test early and frequently because defects detected in earlier development phases are cheaper to fix and result in significantly fewer surprises when the code is actually delivered. Make testing a parallel activity to the development process to make all stakeholders aware of quality objectives. By doing this, testing will be seen less as a bottleneck activity— performed after code is delivered under extreme time pressures—and more as a key contributor in the overall quality process. Since software project success can be directly connected to a solid understanding and articulation of requirements, organizations leveraging RBT place a great deal of importance on the quality of requirements. For example, best practices within the RBT requirements quality process include those DECEMBER 2006

described in “Requirements QC.” 2. Test with your head, not your gut. Most organizations value experienced testers and count on you to catch what others might overlook. While being a single “go-to quality expert” may be great for an individual contributor’s career, it can also put unnecessary pressure on that person and the other members of the team because the organization is relying on instinct rather than collective knowledge to achieve quality goals. Today, test case design is rarely performed in a systematic and rigorous manner. Rather, many tests are designed based on gut feel and intuition, leading to unpredictable quality. Best practices within RBT require organizations to support methodical and systematic test-case design to ensure predictable test coverage. A rigorous, logical test-case design process facilitates test case design that doesn’t depend on the skill or experience of specific testers. Rather, it helps organizations inject repeatability and consistent methodology to the test planning process to make test coverage more predictable. Organizations using this method also can apply various optimization techniques to produce the minimum number of test cases required for sufficient test coverage, making the testing cycle faster and more manageable. In most organizations, textual requirements are very important, given that almost all business analysts and domain experts use natural language to express requirements. However, using textual requirements can make it more difficult to achieve and validate high test coverage. As a result, testers relying exclusively on natural language can use only their intuition to claim that the set of test cases that they designed covers 100 percent of the functionality captured in the requirements. With nonstructured textual requirements, these testers lack a formal way to show complete test coverage, so often “corners” or even major scenarios remain untested until defects occur in production. To systematically achieve high test coverage, organizations must create more formal and structured representations of requirements. Once this is done, it’s possible to define “logical” (skeletal) test cases and ensure optimal coverage of requirements, as these

REQUIREMENTS QC Validation of requirements against business objectives. This optimizes project scope by ensuring that each requirement satisfies at least one business objective. If the requirements and business objectives don’t match, refinement is necessary. Review of requirements by stakeholders. Feedback of users and domain experts should be used to refine the requirements before additional work is done. Development of use cases. Map requirements against a task- or interaction-oriented view of the system. If one or more use cases can’t be addressed by the requirements, the requirements are not complete. Application of language analysis techniques. Detect problematic phrases in the requirements and fix them to guarantee consistency and clarity. A problematic phrase—language that is ambiguous, unclear or inconsistent—can be interpreted in multiple ways by different people, which can lead to confusion and errors later on. Ambiguous terms also result in requirements that are not testable.

logical test cases eventually evolve into the actual set of tests that are run against the system. Multiple techniques can be used to provide structure and formality to natural language requirements. They reveal cause-effect relationships embedded within requirements, expressing requirements as a set of conditions (causes) and resulting actions (effects). Cause-effect charting is one of these techniques. Another way to achieve similar goals is to express requirements as flow charts, since they naturally depict precedence dependency between actions as well as conditional branching of activities. Once expressed in the structured manner described above, it’s much easier to derive the equivalent test cases for the requirements. A set of logical test cases can be defined (manually or automatically), which is exactly equivalent to the functionality captured in the requirements. However, this set of test cases may include many redundant cases (overlap with other test cases). To optimize the number of test cases but still provide full coverage, techniques such as decision (truth) tables www.stpmag.com •

RBT SUCCESS

can be applied if cause-effect charts are used to structure the requirements. If flow charts are used for that purpose, generation of an optimal set of test cases means finding all unique paths on the flow chart for which there are known techniques.

RBT ERRORS Once test cases have been designed, they’re likely to contain errors. To detect and fix them, organizations using RBT best practices incorporate reviews of the test cases by the requirements authors, end users and domain experts. This practice provides key stakeholders with an opportunity to review the requirements and the test cases in their structured form, and catch any defects introduced in the transition from natural language to formal form. To avoid costly rework, test cases also should be reviewed by the developers who wrote the code to ensure that they understand what will be tested and to provide them with a structured view of the requirements. Finally, individual code modules should be reviewed against the structured requirements that trace to them to ensure that the delivered code con-

• Software Test & Performance

forms to the requirements. It’s much easier to match the algorithmic expressions captured in code to the formal view of requirements than to compare them to their nonstructured view. 3. Test with measurement and improvement in mind. It’s often said that what cannot be measured does not exist. Using RBT best practices, organizations can manage and improve quality processes through measurement. Throughout the RBT process, multiple measures can be used to quantify the status of deliverables and activities. This helps managers and process experts track the progress of quality initiatives across the IT application portfolio. Traceability plays a critical role for organizations using RBT, and maintaining traceability information among requirements, test cases and tests is crucial. This information is required for monitoring progress and coverage, as well as for properly managing the impact of changes in requirements. Without it, determining which test cases or tests should be changed if a specific requirement changes is made more difficult. While capturing and maintaining traceability data is very important, many software development organizations find

QUANTIFY REQUIREMENTS Information that should be measured: • Percent of requirements reviewed by domain experts, designers and coders • Percent of requirements that contain ambiguous terms • Percent of requirements with formal representation • Percent of formal requirements covered by formal test cases • Logical and actual code coverage

that it’s extremely difficult to get right. The primary reason is because most tools in the market require the individual practitioners in the software delivery chain to manually create and manage traceability information. This is an overhead that many aren’t willing to accept. To deal with this challenge, strongly consider tools that can automatically manage links between work products. By deploying solid RBT and QA/testing best practices, organizations will be better equipped to maximize the business value of their software through quality initiatives that span the complete software life cycle. ý

DECEMBER 2006

Photograph by J.Hart

Aligning Testing With Its Context By Adam White and Steve Rabin

oftware testing is part art and part science. To be effective, it requires a combination of looking at both the components of a system and the way they

work together as a whole. Mainly, testing requires you to think critically and ask the right questions. Often, testing is a mixture of creativity and defined practices. But with the complexities and time constraints usually involved in software development cycles, it can be easy to lose sight of what’s needed to validate a software product. This brings us to the subject of context. If the role of software testing is to manage confidence and report on risk, the tester must ask the right DECEMBER 2006

questions to move the product in the right direction at the right time. By doing this, testers help improve the quality of the product. As will be seen, testing is very much a context-dependent activity. For veteran testers who might have begun to forget or take certain testing procedures for granted, consider this article a refresher course.

What Is Testing? This question has many possible answers. James Bach, co-author of “Lessons Learned in Software Testing: A Context-Driven Approach” Adam White is manager of test engineering at PlateSpin, a data-center automation software company. Steve Rabin is CTO of Insight Venture Partners. www.stpmag.com •

CONTEXT CONSCIOUSNESS

methodologies and values (Wiley, 2001), and owner of TABLE 1: TESTING LAUNDRY LIST will be vastly different from statisfice.com, a testing conPurpose Type those of the first example. sulting and training company, Your job as a tester is to defines testing as “questionDetermine if regressions are caused and Unit tests estimate thread safety and code coverage determine the testing activiing the product in order to ties that can help you confievaluate it.” Build verification tests Quick “smoke” tests designed to ensure that dently say, “The product is Bach’s co-author and testthe product is ready for further testing ready to ship to customers.” ing guru Cem Kaner, in their Depending on context, entireBlack Box Software Testing Determine if all product functions work as Full functional tests intended ly different testing strategies course, defines testing as a might be appropriate for each “technical investigation of the Can root out interface errors and help with UI tests development project. product under test conducted user friendliness The techniques shown in to provide stakeholders with Table 1 provide a starting quality-related information.” Determine consistent functionality across Regression tests builds point to help you determine (See www.testingeducation when you need to report on .org/BBST.) Ensure that product isn’t overwhelmed by Load tests risk and give project owners In his 2004 keynote at the masses of users or data useful information. It’s up to first annual Software Test and you to develop expectations Performance Conference, Either by build or platform, can help a team Benchmarks set and adhere to performance goals for each of these tests in “The Ongoing Revolution in their particular context. Software Testing,” Kaner said, Verify successful deployment on a variety of Installation tests “Testing is investigation. As operating systems and platforms Change Is The investigators, we must make Only Constant the best use of limited time Over time, projects unfold in ways that and resources to sample wisely from a As an illustration of testing conare often hard to predict. In fact, huge population of potential tasks. texts, consider these two example projchange is perhaps the most dependMuch of our investigation is exploratoects from “Lessons Learned” as posted able constant in software development ry—we learn as we go, and we continuat the www.context-driven-testing and testing. ally design tests to reflect our increas.com Web site. Safety-critical applications. In the Rather than bucking the tide, you ing knowledge. Some, and not all, of first example, you’re testing the conmay as well accept the fact that these tests will be profitably reusable.” trol software for an airplane. In this requirements will always be in a state Context Counts context, “correct behavior” translates of flux and test plans are never comIt’s entirely proper for different test to an extremely technical mathematiplete. So it’s better to spend your time cal formula defined by FAA regulagroups to have different missions. designing and executing tests than on tions. A core practice in the service of one updating test plans. Anything you do—or don’t do— mission might be irrelevant or To keep pace with the project, take a could cause fatalities and be used as counterproductive in the service of lightweight approach to test documentaevidence in a lawsuit. This developanother. tion. Detailing every step required to run ment staff must share an engineering a test will most likely be a waste of time culture of caution, precision, repeatabecause tomorrow, some of those steps CONTEXT COMES OUT bility and mutual review. Such software may not be around. also must pass rigorous certification To create context-driven testing customized for your company, first you must Value-Added Testing procedures. figure out what kind of coverage you Data-safe applications. In the second Good software testing is a challenging intend to achieve. By coverage, we mean example, you’re developing a word intellectual process, so you should the models you have for the product and processor for use online. In this conmake your testing an intellectual activthe extent to which you have tested text, company culture involves storing ity. Two ways to do this are to get against them. user documents reliably and presentinvolved at an appropriate point in the The next step is to determine your ing an interface likely to lure a sizedevelopment life cycle and then push oracles, the principles or mechanisms by able audience away from Microsoft to get test code as soon as possible. which you recognize problems. To mainWord. Throughout the development cycle, tain clear communication among project have no regulatory requireYou stay focused on which part of the cycle team members, be sure the entire team is ments; only those governing your abilyou’re working in and what your role apprised of the project’s scope and can see and understand its overall purpose. ity to compete and make a profit. is within it. If development slips and the release Here, usability, reliability and time to If your role is to automate testing, date doesn’t move, you should provide a market matter more than the softkeep in mind that the practice involves list of features that won’t be tested to the ware’s ability to complete tasks in more than simply automating manual project owners. Their views on what can fewer than 10 nanoseconds. And while tests. Also, testing and automation slip and its impact may help you test this development staff might have tools are important, but are not a more efficiently. come from an engineering culture, replacement for knowing what to test they need not have, and their goals, and how to test it. Test artifacts are

• Software Test & Performance

DECEMBER 2006

CONTEXT CONSCIOUSNESS

worthwhile to the degree that they satisfy their stakeholders’ relevant requirements. Among the final testing phases is acceptance testing, also known as QA or beta testing. This phase, which determines the acceptability of the application and the system in which it operates, often involves the application’s intended users and other members of the project team. To help prevent misinterpretation of acceptance testing results, it’s important to document the context of those results in terms of the people doing the testing and who’s responsible for determining acceptability. Also important is input from the people producing the application under test.

Are We There Yet? Too often, market pressures take the decision to ship from the development team and give it to marketing or other executives. Whether or not this is your situation, here are some helpful benchmarks for determining software readiness. 1. All critical and high-priority bugs are fixed. These issues, identified by the development team or requested by support, product management or other project owners, are repaired. 2. All critical functions of the product have been implemented and tested. A testing matrix is a great way to track this information. It could be as simple as a spreadsheet that contains each feature listed in the left column and each build number across the top. Each cell could have three (or more) status codes: green, red and gray. Green indicates that no issues have been discovered with that feature. Red indicates a problem; in this cell, the bug number is shown. Gray indicates that more testing or investigation must be conducted before a reliable status can be achieved. (Check out the testing matrix posted at www.satisfice.com/ presentations/dashboard.pdf.) The usefulness and priority of the risk matrix can vary during the project life cycle. Near the beginning of the project, interest in your ability to ship the software is generally low, so you shouldn’t send the document out to a broad audience. The week or two DECEMBER 2006

before the release is a different story— in the last stages, interest is piqued, so you should update and distribute the risk matrix on a daily basis. This information can help your test team achieve visibility and can help effect buy-in at all levels of the project. This also prevents your team from becoming the “quality gatekeepers.” 3. Automation reports proper results. If you’re automating your testing project, make sure that automation is focused on the appropriate areas. 4. Requirements are verified. Walk through the requirements and make sure that you have test case coverage for each item. 5. Metrics. A useful metric to start with is the number of customer issues successfully fixed. Bug rates during the project may also provide insight, but only if all variables stay constant throughout the entire cycle (which

• Document the context of your team’s acceptance testing roles to help prevent misinterpretation of results.

• rarely happens). Be sure to provide context to the stats, or the numbers/charts/graphs could be wildly misinterpreted. 6. Test case coverage. This is a dangerous metric even when not used in context with the other heuristics. It’s also risky to continue to aim for coverage when the product is growing and testing resources are changing. Reporting this metric can make those who don’t understand testing feel really good about bad software. 7. Ask your team. A good way to figure out if you’re done testing is to ask your team members directly, “Can we ship?” “Would you feel comfortable giving this to customers?” or “Does this product meet the goals for the

TRACEABLE AND REAL Here are some ways you might provide value: • Make test results traceable to test plans. • Make test results traceable to requirements documents. • Model how software is used with use cases or real-world scenarios.

release?” You’ll be able to gauge their confidence in the product and how it ties back into their own work.

Keeping an Eye on Risk Despite your best efforts, sometimes risk catches up with your project. Such unfortunate events occur in various ways. Maybe your team failed to find a major bug—invariably, your most prized customer finds it after you ship them your latest version. Perhaps you shipped too soon. Shipping software is a trade-off between breadth versus depth of product coverage. This is sometimes the case when the decision to ship belongs not to the development team, but to marketing or other executives. Or, you’ve shipped too late. If you’ve selected the wrong metrics or focused on the wrong pieces of functionality, your progress may stall and release may be delayed. To avoid these pitfalls, keep context in mind as you plan and execute your testing project. By customizing your testing process to accommodate your project’s unique culture and needs, you’ll be able to enhance your chance of success and minimize risk. ý RESOURCES Books • “Testing Computer Software,” Cem Kaner, Jack Falk, Hung Q. Nguyen (Wiley, 1999) • “Lessons Learned in Software Testing: A ContextDriven Approach,” Cem Kaner, James Bach, Brett Pettichord (Wiley, 2001) Online • www.bettersoftware.com • www.satisfice.com • www.developsense.com • www.kaner.com • www.context-driven-testing.com Training • Black Box Software Testing www.testingeducation.org/BBST/index.html • Rapid Software Testing www.satisfice.com/seminars.shtml

www.stpmag.com •

Best Prac t ices

SCM: More Than Code Check-In, Check-Out Media publication, I couldI’m not sure what’s wrong n’t help but notice that these with you software developsame struggles ring true in ers. We writers and journalthe world of code. New tools ists figured out the issue of and technologies promise to configuration management finally slay the configuration long ago. management dragon. Yet Microsoft Word’s Track dealing with change remains Changes feature handles important and, if anything, the editing process for indirequires software engineers vidual articles. And paginawith more skills and training tion and layout software easGeoff Koch than ever. ily deals with changes to the The first bit of advice, say many conactual publication. figuration management pros I’ve interThe fact is, our tools make it easy to viewed, is to pick a tool, any tool, and manage change, and given the mature then encourage, cajole or browbeat the state of the software configuration manteam into using it. It seems painfully agement market, so should yours. commonsensical, but different develBefore you bang out an irate e-mail or opment projects can often lead to the decide to use this issue of Software Test use of a variety of tools. As a result, says & Performance in your birdcage, know Scott Basham, Convergys senior manthat the preceding two paragraphs were ager for developer tools, teams work in an attempt, however lame, at satire. It’s silos and don’t share configuration manjust that after many years in various proseagement techniques, developers find it related jobs, I’ve endured no small numdifficult to move between projects, and ber of tips, assurances and shiny, happy their employers miss out on cost savings presentations about how tools will finalassociated with volume purchasing and ly tame the change management beast license sharing. that stalks most writing organizations. “Companies need to make an effort Yet things never seem to work out to consolidate on a single configuration quite as promised. management tool,” says Basham, whose Taming Change Cincinnati-based company recently impleIncorporating changes to a document from mented Telelogic’s Synergy product. multiple reviewers and editors is a pain, This sounds easy enough, but if you especially when one of them invariably have a moment, visit the list of revision works from the wrong version of the doccontrol software at Wikipedia (http: // ument to begin with. And while pagination en.wikipedia.org/wiki/List_of_revision has seen lots of progress since the days of _control_software)to browse some two typesetting and photographic plates—I’m dozen open-source choices and at least too young to know this for sure, but I take as many commercial offerings. Each tool it on faith—at most of the places I’ve has assorted pros and cons, and with such worked, the designated pagination supera variety of freeware, even a small develuser has always been kept busy by the many opment organization is likely to have code befuddled editors trying to get the tool to bases stored in several different tools. do exactly what they want it to do. In compiling my recent Special Report SCM Is More Than SCM on configuration management for SD After you’ve digested that Wikipedia page, Times (see the Nov. 1 issue), also a BZ consider this second bit of sage wisdom: DECEMBER 2006

Configuration management today is about far more than revision control. The field increasingly entails everything from branching and merging, change sets, build management, continuous builds and integration to defect tracking and deployment, says Mario Moreira, author of “Software Configuration Management Implementation Roadmap” (Wiley, 2004). “A driver of this is that development is becoming more complex, and configuration management seems to be a good place to build on to handle the complexities,” says Moreira, director of technology at a large financial services company in Boston. “In effect, we’re seeing configuration management systems becoming more elaborate as new functionality is added to them.” For someone content to quietly handle a small team’s “check-in, check-out” use of an old favorite like Subversion, an open-source revision control system, the expanding scope of configuration management can sound daunting. Yet there’s incentive to get smart and retrain. Money magazine lists software engineer as No. 1 in its Best Jobs in America feature (http://money.cnn.com/ magazines/moneymag/bestjobs/ index.html) for 2006. And within the category, release engineers—who typically perform configuration management roles, Moreira says—are hailed as having the top-paying jobs, often earning six figures. Even though the reach of configuration management may be growing, this doesn’t necessarily portend the need to wrestle with feature-bloated and CPU cycle–hungry applications. If anything, many companies are seeking to streamline and consolidate all data-center hardware and software. This trend is evident at Google, according a recent blog posting Geoff Koch is a writer in Lansing, Mich. Write to him at koch.geoff@gmail.com. www.stpmag.com •

Best Practices (http://www.robertcowham.com/blog / scm/p4_euro_conf_2006.html) by Robert Cowham, principal consultant with Vaccaperna Systems in London. Blogging from the Sept. 19 Perforce European User Conference in London, Cowham captured statistics about Google’s use of Perforce-based software configuration management, based on a conference presentation by Google’s Dan Bloch. More than 3,000 users at the search company access a single Perforce repository running on a Hewlett-Packard 4-way Opteron server with 128GB of RAM. “Sounds like it wins the contest for largest number of users against a single server!” writes Cowham.

Flexible and Form-Free The top prizes in the contest for most hype in the world of journalism and software likely go to blogging and agile programming, respectively. Both are variations on the idea that flexible and relatively form-free content producers (whether they’re churning out prose or

computer programs) will eventually smash the old process-heavy way of doing things. In journalism school a few years ago, I remember old-guard editors gnashing their teeth about the first experiments with blogging at various newspapers. They grimaced when told how many of these newsroom bloggers would first write and publish to their blog and only later, if at all, would their editors read the posting and make or suggest small changes. The idea of publishing text that hasn’t been vetted or edited was anathema to the ink-stained crowd. In response to such reticence, blogging zealots continue to wonder what’s wrong with traditional media, making assurances that the wisdom of crowds and bottom-up reporting will eventually trump the potential output of any news organization. The landscape is undeniably changing for both mainstream and trade media organizations, but last I checked, The Wall Street Journal has a circulation of more than 2 million, and BZ Media now puts out three publications and a handful of other editorial products, including e-mail

newsletters and various supplements. Both the Journal and BZ Media rely on old-fashioned editors and reporters, so let’s just say that reports of the death of traditional media have been somewhat exaggerated.

Straitjacket—or Solid Rigor? But what about the imminent demise of traditional programming? As rigid process gives way to the freeing norms of agile work styles, configuration management finally will be seen as the stultifying straitjacket that it is, right? That’s another lame attempt at satire, though it does evoke quite a response from around the configuration management community, including from Moreira, who gets the last word this month: “At the end of the day, agile methods allow a team to get release one out there quicker. However, upon embarking on release two, you immediately start needing processes and tools that cater to parallel development, since you immediately need to bug-fix release one while working on release two.” ý

Index to Advertisers Advertiser

URL

Axosoft

www.axosoft.com

Critical Logic

www.critical-logic.com/products/TMXwrx

EclipseCon 2007

www.eclipsecon.org

Instantiations

www.instantiations.com

ITKO

www.itko.com/lisa

Mercury

www.mercury.com/qa/stp

Perforce

www.perforce.com

Seapine Software

www.seapine.com/stpscm5

Software Security Summit

www.S-3con.com

Software Test & Performance

www.stpmag.com

Software Test & Performance Conference 2007

www.stpcon.com

SQS

www.sqs.com

Telelogic

www.telelogic.com/softwaretest_synergy

• Software Test & Performance

Page 2

DECEMBER 2006

Future Future Test

Test

Model-Driven Architecture’s Future Is Now Have you ever wondered the specification, consider why your software developthis: First, by adopting an ment organization has MDD approach you recogproblems meeting its nize the need for a new release dates and quality development methodology; standards? Often, attempts second, the key for success to remedy the situation is the iterative coupling of using external consultants design and verification and weekly status meetings throughout development. just add to the confusion. Adopting the ModelDesigning and building Wolfgang Kaml Centric Mentality complex systems and appliWhat does this mean? Simply purchascations is a challenge. An even greater ing an MDD tool won’t buy quality and challenge is the task of ensuring that all success; it still must be correctly applied of the different processes mapped in a to your software development project. software system have been well tested in Adopting MDD can be perceived by order to predict the application’s permembers of software development formance. organizations as akin to supporting a While testing adds time and cost to rival football team. With MDD, you’re the development process, the expense committing to a shift in which the modof dealing with undetected errors later el becomes the focus, and your intelon is typically far greater, both in hard lectual property is no longer the code, costs and time-to-market. The earlier but the model itself. that an error or performance issue is Taking such a model-centric detected, the cheaper it is to fix the approach provides for success, since you problem. now have a single view of all project One way to help eliminate disastrous requirements, architecture and design, errors early in the lifecycle—before you as well as the means to adopt an overall miss milestones and blow your budget— development process. MDD and a wellis to use model-driven development defined process offer many advantages (MDD). This enables organizations to over traditional software development visually manage their entire develop(a.k.a. hand-coding). ment effort using graphics-based tools. For example, all information stored The Model Driven Architecture in the model is based on a well-defined (MDA) specification, developed by the and standardized notation, allowing for Object Management Group (OMG)—a easy global adoption regardless of your large computer industry consortium— spoken language. The entire project can includes guidelines for combined develmore easily be navigated from beginopment and testing using MDD. MDD ning to end, since all elements are synemphasizes early design testing and tactically and logically connected. error avoidance, helping developers Architects, designers, QA and all others accurately predict and control costs. involved can easily review the project To reap the greatest benefits from

• Software Test & Performance

model—whether they know a programming language or not. Most importantly, tests can be applied to this project model from the earliest life-cycle stages through to final delivery. One way to apply tests to the model is to use an MDD tool capable of “executing” the application; this execution simulates the design’s activity and behavior. During execution, you can record the sequence of messages and signals sent between objects. These sequences can be reviewed for accuracy and efficiency, enabling the reviewer to find errors and suggest design improvements. The model can be automatically reexecuted at a later design stage to verify that it still produces the same expected result. This allows verification of the high-level architectural design very early on, with confirmation that the later, more detailed design has not altered the overall results. With the emergence of increasingly complex service applications, such as those in a Service-Oriented Architecture (SOA), your organization sometimes loses control of key application elements by relying on the performance and quality of external services. You can counter this loss of control by testing these applications early and often. Foregoing testing is a bit like driving a mountain road at night without your headlights. You might get lucky and find the way down… or you might not. Why risk it? We’ve all heard how 80 percent of a software project’s cost is associated with bug and error fixing. If you can eliminate even half of your introduced errors through early model execution and testing, your project is far more likely to succeed. Test and verification through model-driven development is not a future technology; it’s available today. What lies ahead is broad adoption, which will accelerate development and application pervasiveness exponentially. And that’s what we’ll need for the future of testing. ý Wolfgang Kaml has been in the software development industry for more than 15 years, including leading development roles at Porsche and Volkswagen. He is currently a senior product manager at Telelogic. DECEMBER 2006