We (the Sogeti Rosario TAP team) managed to generate test cases directly from Rosario Activity Diagrams and automatically wrote them into Camano!

In previous articles I already wrote about the generating part of this coupling (see Generating Test Cases part 1 and part 2 ).
My wish to write the test cases directly into Camano triggered Clemens Reijnen to dive into the code. Clemens managed to create a "Generate Test Cases" button within the Activity Diagram functionality of Team Architect!
After pushing on this button, the Test Cases pop up like magic in Camano. Ready for use!
See his video about this!

So…..What's next?

I started already some thinking about Unit Testing Support.
Designing test cases for the Unit Test is mostly done manually.

I don't like manual work……I think that this work can be automated to!!

It would be great if we could create another Magic Button!
Preferably: a button that can create "ready for use" Unit Test cases.

Generalist Testers (Manual Testers) do like on-the-fly testing! It feels good to be creative and impulsive!! Let's react on the behavior of the system!........... Maybe it's not as efficient and/or effective as structured testing, but it is fun!

... what about "Non Reproducibility"?

Too much fun brings also disadvantages. In complex systems, with many dependencies under the surface, on-the-fly testers aren't able (most of the time), to write reproducible bug reports. And that's a nightmare for the project manager. Non reproducible bugs are time consuming and expensive, so "on-the-fly testing" is banned out of the life of the Generalist Testers and replaced by structured test methods.

"Reproducibility refers to the ability of a test to be accurately reproduced,
or replicated, by someone else working independently" - Wikipedia


On-the-fly Testing...

Nowadays testers must work in formal structures, of course for efficient testing, but especially for generating reproducible bug reports. Writing an accurate bug report is NOT easy. It takes relatively much time, and even then they are not accurate enough, so developers may call for more information, or even worse, they close the bug report with the state "non reproducible". And believe me...That's not funny at all!

Why do I care?

I care because I want to improve the way testing is implemented in the complete application lifecycle [ALM], and besides that,...it's my job! I am process manager of the Managed Testing Services of Sogeti. If I see chances to improve our services, I go for it!

The new test suite of Microsoft (codename Camano) is in my opinion a great chance for improvement. Instead of converting Generalist Testers into technical skilled testers, Microsoft has chosen to support the way Generalist Testers like to work: "Manual Testing"!

Camano (part of Rosario) is the code name given to the Microsoft standalone testing suite for General Testers. Camano supports planning, creation and executing of manual test cases (CTP April 2008: for testing websites). See the blog entry of Randy Bergeron for some of the latest screen dumps.

Camano fights the non reproducible bugs

Generalist Testers must write accurate bug reports, but now they can stop detailed manual logging of their actions. Because the bug reporting of Camano is great! Camano can keep track of the complete manual (structured or not) behavior of the Generalist Tester. So if a tester is a bit enthusiastic and performs more, better or other tests than originally planned, Camano don't mind, the Microsoft Test Runner records everything in the background for the later uses:

1. Regression testing: the whole script or part of the script.
2. Export to Visual Studio for the creation of automated scripts (to be performed by technical skilled testers before releasing).
3. And for bug reporting!! If a tester runs into a bug, the bug report contains not only all the configuration parameters, it also contains all the steps taken before the bug did occur! Combined with the possibility to capture the window, this support of bug reporting is very strong!!

By combining Camano with the flexibility of our structured test approach TMap®, I can re-introduce on-the-fly testing in our test projects!

Structured "on-the-fly" Testing

Also, with Camano it is possible to have the fun of on-the-fly testing and still report reproducible bug reports. Combining Camano with TMap® makes it possible for us to do result driven test assignments (agreements with the project management concerning time, budget and/or test coverage) and still enjoy testing.
To explain the test teams the balance between structured and on-the-fly testing and how to use Camano in the test project, I've written a fictitious case .
The case contains (a description of):

1. The case specifications: Course Administration application.
2. Creating the basis structure for test coverage
3. The choices concerning freedom versus more structure in the Camano steps

Developers gonna like Camano

I'm sure the developers will like Camano. Especially if they find out that the bugs are reported accurately!
Because: Fast bug fixing is almost as good as making no bugs at all!

Today I read a nice article: "What's In the Box?" by Mary Altman.
This article, about speculative fiction, started with a recognizable explanation of the term "black-box theory":
"The term is typically used to describe a device of which we know or care very little about its inner workings. The entire focus is on its input/output behavior—the result rather than the reasoning."

In the article, also Sociologist Bruno Latour gives a description of the "black-box theory":
"The way scientific and technical work is made invisible by its own success. ... Thus, paradoxically, the more science and technology succeed, the more opaque and obscure they become."
....
Mary ends her article with the conclusion:
"Most simply: it doesn't matter how it works. It only matters that it works."

...and what about the "black-box theory" in the software industry?

I know that many colleague testers believe this conclusion is also valid in the software industry. So, when they organize an Acceptance Test, they rely completely on the testing target: "It only matters that it works". From the perspective of the end-user they are probably right....because the end-user is not interested in (and has no knowledge of) the inner workings. So many Acceptance Tests rely on a complete black-box strategy!

I’m not one of them!

I think the test target: "It only matters that it works" is too thin. It gives no answers for: "When it works, will it keep on working?", or "How about vendor dependency?", or "Is this software maintainable?", or "Are there any hidden features?", and so on...

I think an Acceptance Test must, besides the testing "that it works", find assurance that the inner workings are validated. I agree that it should not be tested in the Acceptance Test, but still it MATTER HOW IT WORKS!!!
So how can we find assurance that it is tested? Is it possible for the software industry to give the Acceptance Test team the assurance the inner workings are validated and correct?

On these questions I am convinced that the testing community can help the developers. I believe, much effort is already done in delivering software of good quality.
But those efforts are hidden and not tuned into an efficient and effective process. Again: I think the testers can help! When testers and developers join together in describing all the quality measures (they already perform), it will be clear if enough measures are taken and/or what additional actions are needed. Chapter 7 of TMap® Next (Development Tests) gives for this inventory a small, but effective model.

The so called "Basic Quality model" describes the quality measures from four points of view:

1. Depth of test coverage;
2. Clarity (description how and when the test coverage is reached);
3. Provision of proof (what deliverables give proof of the agreed test coverage);
4. Compliance monitoring (how does the development process perform its internal monitoring).

Together, developers and testers can fill in these four points of view.

Combined with the quality measures of the requirements and design processes, this gives the Acceptance Test team the opportunity to get a "glass-box" impression how the inner workings are validated.

The glass-box effect gives also the possibility for implementing learning cycles. Implementing the "Basic Quality model" makes it possible to investigate the origin of defects, learn from it and help each other to do it better the next time!....I know...that's a little too optimistic...

So let's do it step by step: Creating a kind of glass-box by giving the Acceptance Testers the possibility to be a virtual witness of the development process would be a great first step!

pictures: www.vuurwerkboer.nl

In a previous post, I showed a simple example of a Rosario Activity Diagram and the Test Cases I directly generated from the XML-file. After the post I got some requests for a better example....
so today I made a more complex Activity Diagram (click on figure 1 to see it bigger).


Figure 1 More complex Activity Diagram (Thanx Angelina!)

And again, I generated the Process Cycle Test cases:
The steps:
1. Get the XML-file from the Activity Diagram,
2. Generate the TMap® Process Cycle Test cases (see the output ).

Do you like it? I Do!!

I don't like manual work. Especially when I think it can be automated!
Last weekend I had a cool breakthrough in my effort to make the work of General Testers less boring!
I succeeded in creating Process Cycle Test cases directly out of the XML file of an Activity Diagram, I created in the Rosario April CTP (Activity Diagrams are part of new Team Architect functionality of the next Microsoft Team System).

Text from TMap® Next :
The Process Cycle Test is a technique that is applied in particular to the testing of the quality characteristic of Suitability (integration between the administrative organization and the automated information system). The test basis should contain structured information on the required system behavior in the form of paths and decision points. The process cycle test digresses on a number of points from most other test design techniques:

• The process cycle test is not a design test, but a structure test: the test cases issue from the structure of the procedure flow and not from the design specifications.
• The predicted result in the process cycle test is simple: the physical test case should be executable. This checks implicitly that the individual actions can be carried out. In contrast to other test design techniques, no explicit prediction is made of the result, and so this does not have to be checked.

TMap® is a registered trademark of Sogeti Nederland BV

Automating the techniques for creating test cases isn't done very often. Most of the effort is pointed at the managing of the test process as a whole and in the recording and executing of test cases. Clemens Reijnen wrote a nice article about this subject in his blog (clemensreijnen.nl). In the last 5 years I did some development on the specification of test cases. I succeeded in creating a set of small web based programs that can support the tester by automating some of the steps during the creation of test cases. But the connection between the Functional Design (f.i. the Activity Diagram) and the tool for creating test cases is done by hand.
UNTIL LAST SATURDAY!

Figure 1 Example Activity Diagram

Last Saturday I managed to make an interface between the Activity Diagram and my tool for creating Process Cycle Test cases.
The steps are very simple now:
1. Create the Activity Diagram( see figure 1) in Team Architect (Rosario April CPT),
2. Get the XML-file from the Activity Diagram (on my disk),
3. Generate the PCT Test cases (see the output ).

Ready for testing? If you want it to test completely manually? Yes!
But I'm not satisfied yet. I want to import these Test Cases into Camano. (Camano is also part of the Rosario CPT; Camano is a tool for automating the manual work of the Generalist Testers)
My first experiences with Camano are positive! The UI is nice intuitive! And I like the support for documenting defects and performing regression tests!

So in the coming weeks I'll try to get connected with Camano!!
(later more...)