Rob Kuijt's Testing Blog
On-the-fly testing with Camano 
Monday, May 5, 2008, 04:20 PM - ALM, Testing, TMap®, Rosario
Posted by Rob Kuijt
Generalist Testers (Manual Testers) do like on-the-fly testing! It feels good to be creative and impulsive!! Let's react on the behavior of the system!........... Maybe it's not as efficient and/or effective as structured testing, but it is fun!

... what about "Non Reproducibility"?


Too much fun brings also disadvantages. In complex systems, with many dependencies under the surface, on-the-fly testers aren't able (most of the time), to write reproducible bug reports. And that's a nightmare for the project manager. Non reproducible bugs are time consuming and expensive, so "on-the-fly testing" is banned out of the life of the Generalist Testers and replaced by structured test methods.

"Reproducibility refers to the ability of a test to be accurately reproduced,
or replicated, by someone else working independently" - Wikipedia



On-the-fly Testing...


Nowadays testers must work in formal structures, of course for efficient testing, but especially for generating reproducible bug reports. Writing an accurate bug report is NOT easy. It takes relatively much time, and even then they are not accurate enough, so developers may call for more information, or even worse, they close the bug report with the state "non reproducible". And believe me...That's not funny at all!


Why do I care?


I care because I want to improve the way testing is implemented in the complete application lifecycle [ALM], and besides that,...it's my job! I am process manager of the Managed Testing Services of Sogeti. If I see chances to improve our services, I go for it!
The new test suite of Microsoft (codename Camano) is in my opinion a great chance for improvement. Instead of converting Generalist Testers into technical skilled testers, Microsoft has chosen to support the way Generalist Testers like to work: "Manual Testing"!
Camano (part of Rosario) is the code name given to the Microsoft standalone testing suite for General Testers. Camano supports planning, creation and executing of manual test cases (CTP April 2008: for testing websites). See the blog entry of Randy Bergeron for some of the latest screen dumps.


Camano fights the non reproducible bugs


Generalist Testers must write accurate bug reports, but now they can stop detailed manual logging of their actions. Because the bug reporting of Camano is great! Camano can keep track of the complete manual (structured or not) behavior of the Generalist Tester. So if a tester is a bit enthusiastic and performs more, better or other tests than originally planned, Camano don't mind, the Microsoft Test Runner records everything in the background for the later uses:
  1. Regression testing: the whole script or part of the script.
  2. Export to Visual Studio for the creation of automated scripts (to be performed by technical skilled testers before releasing).
  3. And for bug reporting!! If a tester runs into a bug, the bug report contains not only all the configuration parameters, it also contains all the steps taken before the bug did occur! Combined with the possibility to capture the window, this support of bug reporting is very strong!!

By combining Camano with the flexibility of our structured test approach TMap®, I can re-introduce on-the-fly testing in our test projects!

Structured "on-the-fly" Testing


Also, with Camano it is possible to have the fun of on-the-fly testing and still report reproducible bug reports. Combining Camano with TMap® makes it possible for us to do result driven test assignments (agreements with the project management concerning time, budget and/or test coverage) and still enjoy testing.
To explain the test teams the balance between structured and on-the-fly testing and how to use Camano in the test project, I've written a fictitious case .
The case contains (a description of):
  1. The case specifications: Course Administration application.
  2. Creating the basis structure for test coverage
  3. The choices concerning freedom versus more structure in the Camano steps


Developers gonna like Camano


I'm sure the developers will like Camano. Especially if they find out that the bugs are reported accurately!
Because: Fast bug fixing is almost as good as making no bugs at all!




add comment ( 111 views )   |  0 trackbacks   |  permalink   |   ( 3 / 3493 )
Testing in the Lifecycle [ALM]... a focus on test coverage 
Sunday, April 13, 2008, 10:53 AM - ALM, Testing, TMap®
Posted by Rob Kuijt
When looking at Testing and more specific Test coverage in the Lifecycle [ALM] you can conclude that much effort is done to test as good as possible, but nobody can tell you what Test coverage is achieved in the successive stages of the ALM.

Work must be done in the thinking and communication concerning the quality levels that should be reached! It is proved to be very difficult to choose the thoroughness of testing and it is proved to be even more difficult to explain the executed Test coverage to the colleagues of the next test levels.

With the appearance of the chapter 14 "Test Design Techniques" in TMap® Next , there is now some light on the horizon. In the "old TMap®" the Test coverage was expressed in terms of dynamic and static quality characteristics, coupled with test techniques, which nobody understood. Even the full-time testers had trouble understanding it through and through.

With the introduction of TMap® Next the test coverage is, more friendly and intuitively, expressed in terms like paths, decision points, CRUD (coverage of the basic operations), checklists, and so on...
Now we can explain the chosen Test coverage practically in plain English!

I can give an example how we introduced this type of Test coverage expression in ALM in a project I've done within Sogeti. In the project we planned a serial of 5 successive test levels: Unit Tests (UT), Component Integration Tests (CIT), Technical End-to-end Test (EET), Functional Acceptance Test (FAT) and User Acceptance Test (UAT). Instead of designing the tests on an individual bases we created one overall "tuned" test strategy.



Picture: Clemens Reijen; from his article:
Testing in the Lifecycle [ALM]... a focus on automation


This overall test strategy was designed in three layers:
  • First; for all the test levels we determined the Basic Quality level, which can be seen as the absolute lowest level of Test coverage (labeled: Bronze). Formal escalation is needed to escape from this Basic Quality requirement. And of course the depth of testing is expressed in terms of chapter 14 of TMap® Next.
  • Second ; based on the BDTM-approach (see chapter 3.1 in TMap® Next) risk classes are determined for each combination of characteristic and object part. Characteristic= what must be investigated; Object part= what must be tested. The Test coverage above the Basic Quality level is, for all test levels, determined in a so called Master Test Plan. In my experience it is easy to communicate with stakeholders when the higher Test coverage levels are labeled as Silver and Gold and even Platinum. And again the labels silver, gold and platinum are expressed in chapter 14 terms.

  • Third; we introduced the Learning Cycle. Every time a blocking or costly defect occurred we analyzed the defect and if necessary we modified the Test coverage definition of the test level where the defect should be found.
    Another example of working with the bronze, silver, gold and platinum labels can be found in chapter 7 "Development Tests" of TMap® Next.


ALM (Application Lifecycle Management) regards the process of delivering software as a continuously repeating cycle of inter-related steps: definition, design, development, testing, deployment and management. Each of these steps needs to be carefully monitored and controlled [Wikipedia].
For more definitions see the article about ALM Definitions in the blog of Clemens Reijnen.

TMap® (Test Management approach) is a registered trademark of Sogeti Nederland BV

add comment ( 51 views )   |  0 trackbacks   |  permalink   |   ( 3 / 3703 )

<<First <Back | 1 | 2 | 3 | 4 | 5 |