Rob Kuijt's Testing Blog
http://robkuijt.nl:443/index.php
Model Based Testing (part 1)
http://robkuijt.nl:443/index.php?entry=entry100627-081107
Model Based Testing is a powerful concept for testing information systems, from which, despite the increasing complexity, more quality and flexibility is expected. It is a methodology that fits well with companies with experience in structured design and testing. Model Based Testing will also adobe.lovers.com support and lower the threshold for companies who want to make a first step towards improving their test basis towards a better, faster and cheaper test process.
The core of Model Based Testing is the use of tools to first, create the actual design and secondly to compare the simulation of the process and predicted outcome of the test generation with the requirements at an earliest possible stage. Thus, the various "paths" through the procedures / processes including related business rules are tested and checked much earlier. Subsequently, automatic test generation creates efficient and reliable high quality test cases which excel in compactness and low cost.
Model Based Testing helps to improve the overall development process: the impact of demands and wishes becomes clear in a fast manner, design errors are found at an early stage, debugging code caused by wrong assumptions and interpretations decreases and reuse of designs in new projects is easier. In short, developers achieve greater results in less time and at lower cost.
The tools needed to support Model Based Testing are available and are developing rapidly based on the experience of cutting edge companies like Sogeti. It becomes increasingly easy for companies to make the choice and take advantage of Model Based Testing.
Rob Kuijt ]]>History of Test Automation
http://robkuijt.nl:443/index.php?entry=entry100110-214414
Test Automation is the use of software to control the execution of tests, the comparison of actual outcomes to predicted outcomes, the setting up of test preconditions, test design, and other test control and test reporting functions. Commonly, test automation involves automating a manual process already in place that uses a formalized testing process. (Wikipedia)
Working with Model-Based Testing I get many questions on “Test Automation”. To be honest, around a year ago, I could not answer them at all. After a year playing with “test industrialization” my knowledge is better. But still, especially the timeline (“when was a specific type of test tooling introduced”) was a gap in my mind. An overview: “History of Test Automation” would be very practical.
On internet, I couldn’t find the complete Test Automation timeline I needed, so I started creating one…
Inspired by the powerpoint slides: “Generations of Test Automation” by dr Mark Utting, I made a slide show (kind of e-learning component) which gives an overview how the (automating of the) test process did evolve since the 1940's. I created some nice graphics, added some definitions and expanded the presentation with more detailed slides and background information from TMap®, Wikipedia, James Bach, William E. Lewis, Gunasekaran Veerapillai and Marvin Niven (thanks guys!) and uploaded the slide show/e-learning component for future use… Interested? See “History of Test Automation”
]]>Test Case Design: Manually or Automated?
http://robkuijt.nl:443/index.php?entry=entry091220-074237
There are several tools that can help with the creation of test cases. So why don't we succeed in automating the Test Cases Design Process? In this short article I will explain that, if test collaborates with design, we can make huge progress on this topic!
4 Examples of Test Case Design Let’s imagine a business process that is documented by the design team on one page. Test will create the test cases. In the first example the description is done in plain text. In that case it is not possible to automate the test, and also it is not possible to use a formal test specification technique:
Example 1: Test Design from plain text: many interpretations and assumptions...
Creating test cases from plain text may look easy, but when you ask 5 test engineers to create the test cases you get 5 different sets without any insight in the quality of the coverage. Choosing the process flow diagram technique for describing the business process, will result in less interpretations and assumptions, therefore in much better test cases:
Example 2: Manual Test Design: Process Cycle Test (TMap®)
The advantage of formal Test Case Design techniques like Process Cycle Test (PCT) is that it can be automated! How? The first step of deriving test cases with PCT, is to identify the paths and path combinations within the process flow diagram. Step 2: Instead of manually combining those path combinations to test cases, the path combinations (joined with a short description) can be inserted in a Test Design Tool. Step 3: The test cases are generated. Comparing with manual test case design: much faster; less knowledge is needed, but you need a tool* (license):
Example 3: Using a Test Design Tool: much Faster!
Most of the time, designers make these process flow diagrams in modeling tools (like MsVisio, Protos, Aris, BiZZdesigner,..). A common feature of the modeling tools is exporting the models in XML-format. Model Based Testing tools* do read XML! So, let’s skip all manual steps and generate test cases directly from the process flow diagram:
Example 4: Model Based Test Case Design
Model Based Test Tools* can generate test cases within minutes. Of course, before using the test cases you must do a sanity check to confirm the tool understood the model correctly but nevertheless the time can’t be beaten manually.
Collaborate! Test Case Design can be automated (I’m already working with these tooling on a daily basis). But, besides getting the right tool(s), there is an important condition: the Test Base must have a minimum level of quality. For instance: instead of describing business processes in plain text, they should be specified with the help of activity diagrams or process flow diagrams (and that is not (yet) common knowledge within the design processes). To get an automated Test Design process, let’s join the forces: Together, Design and Test can make projects much faster and cheaper!
*) Example of a Test Design/Model Based Testing Tool I personally often use: STaaS/COVER (Sogeti Netherlands)
]]>ALM and MBT
http://robkuijt.nl:443/index.php?entry=entry091019-190829
Both ALM (Application Lifecycle Management) and MBT (Model Based Testing) are hot. I'm pretty busy running from customer to customer starting/supporting MBT proof of concepts and assignments. With some explanation all my contacts are convinced that the combination of ALM and MBT is THE roadmap for the future.
The advantages of ALM can already be implemented by choosing for the new generation of tools of Microsoft (VSTS2010) and IBM (JAZZ etc.) are presenting to strongly support the collaboration of the different roles in and around the ICT. Model Based Testing has not yet that same level of maturity. Most participants in the Model Based Testing world are very ambitious: They try to Generate Test Cases for direct automatic execution, based on test specific models.
I think this is a bridge to far for most companies using for instance mainly ERP-software. In the ERP-scope most of the testing is still done manually, and jumping to this ultimate form of Model Based Testing (Generate Test Cases for direct automatic execution) will be very tricky and/or costly. That's why we (Sogeti) are introducing also a basic form of Model Based Testing (STaaS/COVER), using the same models as the developers do.
"MBT: ...a bridge to far..."
MBT: Generate Test Cases for direct automatic execution Does this mean that Generating automated tests is a dead end? NO! It is a very good approach for testing high risk functionalities, which are to risky or to complex to test manually. Making test specific models is a costly business and top designers/test engineers are needed to make it work. So be sure to pinpoint these heavy coverage and tooling only on the high risk areas. AND!! Do not try to make the test specific models in the Testing Silo! Look for collaboration with the design department to get these test specific models in the same configuration mnagement as the models the devlopers use to build the software. Preferably in an "ALM Center Configuration Management".
MBT-basic: Using the same models as the developers do On the other hand, when the ambition of Model Based Testing is lowered to only generating the test cases, it is a complete different story. Especially the generation of the (more global) logical test cases can be done from the same (functional and/or requirement) models the developers use as a bases for creating the software. For instance process flows (activity diagrams), decision tables and pseudo code are a perfect basis for generating (logical) test cases. Enrichment of these models (in collaboration with the design department) will bring us to the next step: Generating the physical test cases. In that case the Test Cases are no longer Configuration Items (costly maintenance!) but Test Cases become Work Items (if desired: directly generated from the same models the developers use to build the software).
Rob Kuijt ]]>ALM: the end of the whispering game
http://robkuijt.nl:443/index.php?entry=entry091017-203506
Who don't know the whispering game? For me it was one of my favourite games in my childhood. Whispering a tricky word in the ear of the child next to you and waiting for the twisted result at the end of the cycle....
Nowadays not only children do play this game. In the ICT world practically everyone is playing this game. Only they invented a new name for it: The Waterfall Model .
ALM (Application Lifecycle Management) makes an end to the twisted results of this widely implemented whispering game. Instead of working in Silos with their own project and configuration management, in ALM the requirement, workflow and configuration management is organized in the center of the process. So everybody is working on the same artifacts.
In ALM everbody is working on the same artifacts. No whispering risks at all.. ;-)
And from my archive.. Some famous twisted results of the whispering game:
]]>Model-Based Testing is not "Only Fun" anymore....
http://robkuijt.nl:443/index.php?entry=entry090620-140924
For several years now, Model-Based Testing is mainly used within the (technical) development processes. In the Black Box testing community it was still "just a theory", and in my case: fun to play with. Today is different! I'm completely into Model-Based Testing nowadays. MBT turned, for Black Box testers, into serious business!
Wikipedia: Model-Based Testing is software testing in which test cases are derived in whole or in part from a model that describes some (usually functional) aspects of the system under test (SUT).
For quite some years I'm working with a tool called COVER. Initially COVER was build to help lazy testers, like me, to avoid boring manual activities…and it worked! Using this tool I could easily amaze my surrounding with stunning Speed and Quality. Especially deriving test cases from formal specifications like pseudo code or activity diagrams was fun to do; instead of "solving a boring puzzle" time, it was done in seconds (as a matter of speaking). However, most of the testers in my surrounding didn't recognize the ease of this way of working (or were not lazy enough?) so COVER didn't have much users.
Today is different. The attention for Model-Based Testing is growing rapidly. Previously, when a project had to make test cases, it was accepted that it was done manually and, everybody knew, it should take quite some time. Today the budgets are lowered...What to do?...Less productivity? My suggestion: When you've less money to spent, be creative to get the same results!
First Model then Test!
Model-Based Testing is a nice example of creative thinking! Within a few months the relatively unknown possibilities of Model-Based Testing tools became very popular for Black Box testers within Sogeti. Especially the collaboration-part (designers working together with testers) of Model-Based testing is an eye opener. Everybody knows that it is smart to find defects as early as possible, but it is very difficult to motivate teams to perform good design inspections. Now MBT tools like COVER can do a big part of those intensive inspections, because using the models for generating test cases will directly show the defects in those models in a very early phase of the project! And beside that, it generates a big part of the test cases!
So, when the Designers create models that COVER understands, projects will perform Better, Faster and Cheaper! ....and it works! Some (early) successes from our proof-of-concepts and implementations:
The amount of time needed for creating test cases dropped 50% (and sometimes more);
The maintenance of the (regression) test cases became in average 70-80% cheaper;
And maybe most important: We have proved to find defects much earlier than the project did manually!
Collaboration will change the world of testers! Using MBT is in fact collaborating within the Application Lifecycle [ALM] supported by standardization and automation. Many defects, found in functional testing, have their origin in misinterpretations and assumptions of the requirements and/or specifications. The early usage (generating test cases) of formal models, just after or during the design phase, will create a much more solid base for software coding. For the testers this will mean less and less defects, so besides the generated test cases, the improved quality of the software will also fasten the test process. (PS. When both software and test cases are generated from the same models, be sure that the test objective is clear!)
This industrialization can’t be stopped anymore. The time of creating most test cases by hand has had its peak. Of course, not all test cases can be generated, and also manual testing itself will never disappear, but handcrafted test cases will be less and less common practice. The work of Black Box testers will, in my opinion, shift in two directions: 1) Earlier in the life cycle: joined modeling and helping and supporting development to find defects as early as possible (Master Test Plan consultancy), or 2) the coming market End-to-End testing: the importance of complete chains of applications is growing rapidly. End-to-End Testing will become a separate specialism.
Examples Back to Model-Based Testing: To give an idea what type of models can be used, and how test cases are generated, I've made a some slides (with examples) about the Sogeti MBT tool: COVER. Look for yourself if MBT will affect your work!
It did affect mine! ;-) Rob Kuijt]]>Visual Studio Team System 2010 - Episode 4: Quality Check
http://robkuijt.nl:443/index.php?entry=entry090424-100409
In this episode I will discuss the different practices around the Quality Check in order to do this important check for ALM as efficient and effective as possible.
In the last episode Clemens talked about the support VSTS2010 will give on the collaboration at artifact level. Different roles in the lifecycle work together on artifacts. Each of them adds their knowledge, vision and ideas to the solution from their view point. These artifacts are accessible by every role in every phase of the project, adding value throughout the lifecycle. People are enabled to collaborate, making applications together, and not only by telling what they are doing but most important by working seamless together on the application.
TMap® Quality Check A subsequent measure for increasing the quality of the developed artifacts is an evaluation activity: for instance the review.
The review is a method of improving the quality of an artifact by evaluating the work against the requirements and/or guidelines and subjecting it to peer review.
The review on the requirements and/or design can be carried out as a static test activity before the coding starts. In the review, the following points can be checked, independently of the set requirements: 1. Has the artifact been realized in accordance with the assignment? For example, are the requirements laid down in the technical design realized correctly, completely and demonstrably? 2. Does the artifact meet the following criteria: internally consistent, meeting standards and norms and representing the best possible solution? ‘Best possible solution’ means the ‘best solution’ that could be found within the given preconditions, such as time and finance. 3. Does the artifact contribute to the project and architecture aims? Is the artifact consistent with other, related artifacts (consistency across the board)? 4. Is the artifact suitable for use in the next phase of the development?
Why? Like testing, the Quality Check is a measure to provide insight into the quality of delivered products and the related risks when taken into the next phase of the lifecycle. If the quality is inadequate timely measures can be taken, such as rework by the designers. However, there is never an unlimited quantity of resources and time. In theory it is important to relate the Quality Check effort to the expected risks. A pragmatic approach to determine the Quality Check effort is to look at some past projects and answer the question:
"How many defects, detected in Acceptance Tests, could have been found much earlier, if we had done a Quality Check? (According to the above points)" In my experience 10-20% of the defects could have been found much earlier in the lifecycle when the Quality Check was done properly. And because defects in the Acceptance Test are quite expensive, you only have to find 1 or 2 serious defects in the Quality Check to make it economic worthwhile. So my strong recommendation is:
7 Hints and tips 1. Performing a good Quality Check is a kind of inborn specialization. Find a person who is good in recognizing texts with high potential risk on assumptions and or interpretation errors. In other words: find a pencil-pushing, nit-picky quality geek! ;-) Most professional testers are proud to have these qualifications.
2. Before checking an artifact: What is the quality of its source? Is the source of the AUC (Artifact Under Check) ready? authorized? stable? If not, consider to do also a check on the source of the artifact to quantify the possible changes in the (near) future,
3. If a previous version of the artifact is available, and the quality of that version is known: make use of a so called Comparison Tool to find and check the differences. AND! Check always the consistency of the “change register”, especially when the “change register” is used in the next phase to implement the changes.
4. Combine the Quality Check with the estimation activity for the next phase. If the estimation is done by another person, let them work together!
5. Use a checklist as a reference! For your own protection, a checklist prevents that too much attention is paid to the use of standards and correct spelling or even to these aspects alone (This can be a cause of friction among the various people involved.) Partly owing to the diversity of design techniques and information sources that, it is not possible to create one general checklist per artifact type. Therefore, checklists should be created specific to the situation per organization and per project. Of course you can use examples like Testing Requirements or Testing Use Cases as a start for the creating of your checklist.
6. Make always clear what checks you will perform. By communicating your checklist you can prevent a lot of misunderstandings on later findings.
7. Audi alteram partem (hear the other side). Don't report on findings/defects without a fair hearing in which the author of the artifact is given the opportunity to respond to the "accusations" against his work.
Collaborate For some, it is very tempting to do a review in their own silo: Get the stuff to check, find as much as defects as possible, receive applause for the prevented damage and do that over and over again…… Wrong! Don’t be a Scrooge! (see my blog: "Does Scrooge exist?") An essential part of the Quality Check is the Learning Cycle. By performing Quality Checks the quality of future AUC’s (Artifacts Under Check) must grow. So collaborate with the designer, information analyst or whoever made that artifact, and get your applause on the better quality of artifacts instead on the number of defects. In the next episode Clemens will explain how the tools support the Quality Check, as well as the collaboration around it, to get the optimum results.
]]>Visual Studio Team System 2010 – Episode 2: No Risk No Test
http://robkuijt.nl:443/index.php?entry=entry090210-200830
In episode 1, Clemens introduced the focus of Visual Studio Team System 2010 on the collaborative effort between the tester and the other roles in the application lifecycle. In this second episode I will start with a short introduction on testing.
Testing is not an aim in itself! Testing is, in fact, a balancing act. What risks must be covered, what results are to be delivered and how much time and money can be spend, based on rational and economic grounds?
The right test strategy will balance among risks and costs .
Testing supplies insight in the difference between the actual and the required status of an object. Where quality is roughly to be described as 'meeting the requirements and expectations', testing delivers information on the quality.
In this, there is no difference for developers who are testing, specialist testers during system testing or generalist testers during the final acceptance test. Choosing the right test strategy is a joined effort between every tester and the other roles in the application lifecycle. This collaboration is needed because, beside insight in the business risks, much (technical and test) knowledge is needed to find the most important defects as early as possible at the lowest price.
The collaboration doesn't stop after choosing the best test strategy. After designing the test cases, also test execution must be organized as a joined effort. Collaboration between the different roles in the application lifecycle is not self-evident. By nature, it looks like developers and testers don't (want to) understand each other. Having this attitude, it is difficult to find out you need each other to deliver software of good quality.
...don’t (want to) understand each other.
The consequence of this virtual (and sometimes real) wall is calamitous for quality. Many bugs arise by doing wrong assumptions and interpretations of the, in generally, unclear specifications and requirements. Reporting, analyzing and resolving these bugs take a lot of time, especially when they prove to be not reproducible or wrong.
An Example: After a development project of 5 man year, testing is done by a team of generalist testers. With much enthusiasm the test team executes the test cases they designed during the building of the system. After 3 months the verdict is given: Negative release advice, because of (8) blocking and (22) big defects, the test team gives the advice NOT to go into production. As you may understand this was a big disappointment for both the project team as the business department. A taskforce was established to keep the delay (damage) in control. After the defect analysis the general feeling became better, after filtering the test faults (20%), the functional wishes (20%) and changes (not passed through to the testers; 30%) the repair costs were approximately 200 hours ( for the remaining 30%). In fact the development team had done a great job. Better communication from both sides, before and during test execution, probably had made the verdict a, celebration worthy, positive release advice. Much, much better than this cold shower!
The above paragraphs show that the collaboration between the tester and developer is important for success. If the collaboration between the different roles in the application lifecycle is not actively stimulated and facilitated, the proverbial wall will arise.
In the next episode Clemens shows us how Visual Studio Team System 2010 will support and stimulate this collaboration... ]]>Testing or Documenting?
http://robkuijt.nl:443/index.php?entry=entry090112-222821
How sure is a company about the quality of their Calculation Component(s)? Are they bug free? Even after, for instance, nine changes? Testing changed components is tricky, because you never get the time to test a Calculation Component with lots of values, because most of the testing is still done manually.
There are many test tools that could do this job much faster, but they are complex to use and mostly pretty expensive. So I did some thinking…….I want to do lean and mean Calculation Component Analyzing! Preferably: User-friendly, quick, low cost and in a way that the outcome is easy to check!
Personally I like pictures instead of figures. So I started some PHP programming, and voila these are my results: The first version is a service that can analyze a “one parameter” Calculation Component. Let me give an example. The test object is a web component with one input parameter. In my home made analyzer service I give the URL of the test object, the start input value, the end input value and the step size (see fig.1)
Fig.1 Input screen of my home made Analyzer
After sending the input, the analyzer performs, in this case, 21 calls to the test object, and responses with the following graphics:
Fig.2 Output screen of my home made Analyzer
This output gives a first impression of the calculation. It looks like: output=input*input (if input > zero) or output=input*input (if input >= zero).
But to be sure, let's make the steps in the range smaller(step=10):
That’s strange! The input value 10 gives a response: zero!
Again, let's choose step=1:
Now I'm pretty sure that the calculation function is: output=input*input (if input greater than ten)
And for a last check (in this example) I choose step=0.1:
Yes, it still looks like: output=input*input (if input greater than ten)
Conclusion: I think that this kind of functionality is valuable, not only for testing, but also for documenting Calculation Components. Above that, it is:
Easy to use (one simple input screen),
Low cost (I did build the function in 2 hours (with the help of open source: PHPLOT)),
Quick (the above analyzing took me 3 minutes (including capturing the pictures for my blog)),
Simple to read.
And it's FUN to play with! Now I'll try to find some use for this kind of functionality, and l start thinking how to handle (and present the outcome) for a “two parameter” Calculation Component.
Rob]]>Does Scrooge exist?
http://robkuijt.nl:443/index.php?entry=entry081126-201908
Fiction or reality? On a, further nice, kind of conference meeting, I did met Ebenezer Scrooge! I didn't know that such Scrooge-type testers do exist!
:-((
Scrooge was fully focused on finding bugs, and if I say fully, I mean FULLY ! Scrooge was completely focused on his own world, collecting as much as bugs as possible, and enjoying it with a kind of scary laughter..... I must say his test results seems to be excellent, very fast in creating test cases, and he knows and uses more test techniques, I ever did. And he has, as we say it, a nose for finding bugs. But his eyes did spit fire when I suggested helping the developers to find the bugs earlier in the cycle. "Why should I destroy my own work" Scrooge replied..... I must confess, I didn't have a response right away. I know that most of the (good) testers enjoy finding bugs. But such a fanatic ego-centric type was new and a complete surprise for me. In fact, I think that Scrooge-type testers, especially this fanatic, are a disgrace for the test profession I love.
So I tried to convince him to change his attitude. Of course I didn't succeed the remaining 15 minutes we met. Probably he needs a visit from the ghosts of the past, present and future!
;-)
For the Scrooges among us For the Scrooges in the test world I have a message: Finding bugs can deliver applause from your surrounding and it may look that your manager is pleased with the extra test hours (he can send a bigger invoice). But in the long term no one (beside you) is happy with a Scrooge attitude. The business users don't get their systems on time, the developers won't help you if you need them, and if the project manager becomes aware of this attitude, he will kick you out (so your manager can't send any invoice at all). So broaden your small ego centric world! Adopt Application Lifecycle Management [ALM] and find bugs as early as possible in a collaborative driven attitude.
For the non-Scrooges How do we fight this irritating phenomenon? In my opinion, the best testers are the ones helping (actively) the developers to build better software. Luckily some of the big test gurus of this world preach the same opinion (see the entry measuring testers in the weblog of James Whittaker).
Wild thought Can we make testers responsible for the quality of the software? For instance: Is it possible to award testers depending decreasing defect rates. I think a kind of Collaboration-bonus awarding mean time between failures and/or decreasing defect rates can work! Has someone a suggestion for the formulation of such a performance indicator or Collaboration-bonus?
I'll look for some like-minded friends and give it a try in the coming period. I’m sure this entry will be continued.... And again: Suggestions are very welcome!