1、Theme,Test automation can slash the cost of text execution, but not all test automation initiatives are equally successful.,Definition,Test automation is the creation and execution of scripts that run functional test cases and report test results without human assistance.,Why test automation?,The pr

2、imary goal of test automation initiatives is to reduce the amount of time and effort required for test execution and, especially, test re-execution. The secondary goals of test automation initiatives are to reallocate this time and effort to: Release software sooner Increase test coverage Pursue oth

3、er avenues to achieve development life-cycle maturity Some combination of the above (most common),The basic economics of test automation,For example: If a test script will be run every week for the next two years (104 times), automate the test if the cost of automation is less than the cost of manua

4、lly executing the test 104 times.,Cost of test automation + cost of automated test execution,Cost of manually executing the test as many times as the automated test will be executed,Automate if:,Building a business case for test automation,Business case built on reduced test effort: Assess the curre

5、nt size of your regression test suite. Determine what proportion can be automated. Calculate savings by subtracting the cost of automation from the cost of manual execution.,Business case built on increased test scope: Estimate number of additional test cases to be executed in the time saved by auto

6、mation (x). Estimate number of production defects avoided per test executed (y). Estimate cost of each production defect (z). Calculate estimated cost savings through avoidance of production defects (x*y*z).,Where is the best ROI for test automation?,Test automation is most valuable for tests that:

7、Are relatively consistent from release to release Replace frequently executed manual tests Verify the most critical functionality Are the most likely to expose defects Cant be executed manually Can easily be data-driven Can easily be keyword-driven,Issues with automation methodologies,Code-oriented:

8、 powerful but inaccessible Requires programming skills that testers rarely have GUI-oriented: short-term gain, long-term pain Results in brittle test scripts because GUIs often change Keyword-driven: efficient and pragmatic Requires the collaboration of resources with different specialties Autonomic

9、 test scripts: Facilitates maintenance through linkage of code and test Assisted manual testing: Offers lower rewards than automation but lower risk as well Not every tool offers equal support for every test automation methodology.,Keyword-Driven Testing Parameterizes Values And Actions,March 1, 200

10、5, “Keyword-Driven Testing Involves New Constituencies In Automated Testing” Forrester report,System integration tests,Accumulate automated test suites from the inside out,Most test automation targets system and system integration testing. But test automation should start earlier with automation of

11、unit and component tests. These types of tests are even more powerful in combination. A mixed suite of these tests permits correlation of failures at different levels to facilitate defect resolution.,System tests,Component tests,Commercial tools for test automation,“Modern” tools: Borland SilkTest C

12、ompuware TestPartner Empirix eTester HP QuickTest Professional IBM Rational Functional Tester,“Legacy” tools: Compuware QARun HP WinRunner IBM Rational Robot Service-oriented architecture (SOA) testing tools: Parasoft SOAtest MindReef SOAPscope Tester iTKO LISA IBM Rational Tester for SOA Quality HP

13、 Service Test,Test automation tools are part of functional testing solutions,May 31, 2006, “The Forrester Wave: Functional Testing Solutions, Q2 2006” Forrester report,Mercury (now HP) offers the leading functional testing solution,May 31, 2006, “The Forrester Wave: Functional Testing Solutions, Q2

14、2006” Forrester report,Go online to customize this Wave to suit your needs,May 31, 2006, “The Forrester Wave: Functional Testing Solutions, Q2 2006” Forrester report,Open source tools for test automation,Open source testing tools: Tend to be better for developers than for testers May be suitable for

15、 testers with scripting skills Are often more oriented around APIs than UIs This makes them more suitable for automating tests that target a more rapidly changing application.,Examples include: Selenium Watir FIT/FitNesse Samie Marathon . . . and on and on Many other open source testing tools are av

16、ailable for unit and component test automation; those listed above are for functional test automation.,Automate more than just test execution,Build,Environment set-up,Snapshotting,Test execution,Storage/ teardown,Check-in,Results logging,Defect repair,Realm of automation opportunity courtesy of tool

17、s for build management, test management, test lab management, and deployment,Case study: the power of test automation plus test environment management,Healthcare company tests changes to desktop apps before deployment to 30,000 workstations. Development cycles were two weeks, but testing wasnt done

18、for another two to three weeks. Regression suite addressed 40 critical applications out of 3,000 total apps and took five days to execute. Functional test automation with HP Quality Center reduced test cycle to three days while maintaining scope of testing. Virtualization and test environment manage

19、ment with Surgient VQMS reduced test cycle to 4 hours.,Road map for test automation,Shore up test management before tackling test automation. Identify candidate test cases for test automation. Assess current skills and supplement if necessary. Select a test automation methodology (e.g., keyword-driv

20、en). Select tools that support this test automation methodology. Align roles and responsibilities accordingly. Proceed incrementally. Measure returns along the way.,Metrics for measuring the success of test automation,Total test coverage Time required to execute test cases Number of bugs caught through test automation Number of executions before test script requires maintenance Time spent on test script maintenance Time spent on test script development,Carey Schwaber +1 617.613.6260 ,Thank you,Selected bibliography,December 13, 2007, “Best