Application Lifecycle Management [ALM] should ensure that an organization experiences an improved "business as usual" in the event of the implementation of new and/or changed functionality. Other (older) industries can give assurance, so the ICT industry should follow (soon?). This article will, I hope, give some next steps towards adulthood, by giving [ALM] directions how to prevent costly or even lethal disasters caused by bugs.

Business Continuity has for [ALM] two viewpoints:

• [ALM] is part of the interdisciplinary concept, called Business Continuity Planning (BCP), used to create and validate a practiced logistical plan for how an organization will recover and restore partially or completely interrupted critical function(s) within a predetermined time after a disaster or extended disruption (see wikipedia for more information),
• [ALM] must give assurance that developing and/or changing applications won't create disasters or extended disruptions! (I won't explain what software bugs can accomplish...).

In this entry I will give some thoughts concerning the second viewpoint:
How to prevent a major disruption caused by the implementation of a new or changed application?

Of Course, the Acceptance Test is very important in our "battle" to prevent the worst to happen. But, as I stated earlier in my article "What's in the Box?", the Acceptance Test alone is not enough! Based on the Business Risks (Business Driven Test Management *) the Master Test strategy in [ALM] must contain at least the following three pillars:

1. Finding Bugs AEAP (As Early As Possibly)
2. Bug Prevention by Testing Requirements and Use Cases
3. Gathering and Analyzing data to do Business Continuity Predictions

*) Business Driven Test Management gives client grip on test process, uses client language, delivers appropriate test coverage on right spot and makes test results visible for client (see TMap.net for more information)

Pillar 1: Finding Bugs AEAP (As Early As Possibly)
Finding bugs as early as possible prevents changes in the software in the last phases of a project.
Recently I participated in an (early) evaluation of an organisation, where the strategy of finding bugs as early as possible was implemented for the complete software engineering department by choosing the Quality Levels, tuning the Test Coverage of the successive test levels and introducing Learning Cycles. They had geat results!
See two earlier articles on this topic:

• Testing in the Lifecycle [ALM]... a focus on test coverage,
• Finding Bugs AEAP (As Early As Possibly).

Problem solved? No, not yet, by diminishing the bugs, it became clear that the number of late change requests and/or wishes for extension of the functionality were disturbing the implementation of the applications. So it became obvious that the next pillar became more important...

Pillar 2: Bug Prevention by Testing Requirements and Use Cases.
If software development is based on inaccurate requirements, then despite well written code, the software will be unsatisfactory. No doubt that the users do want to change the application before it is implemented. Changing an application in the last stages of a project will generate huge risks.


Testing Requirements
Testing the requirements in the early stages of the project will minimize the changes on the application just before implementation. To give the testing of requirements a head start, I derived a checklist from the article "An Early Start to Testing: How to Test Requirements (Suzanne Robertson)"

1. Does each requirement have a quality measure that can be used to test whether any solution meets the requirement?
2. Does the specification contain a definition of the meaning of every essential subject matter term within the specification?
3. Is every reference to a defined term consistent with its definition?
4. Is the context of the requirements wide enough to cover everything we need to understand?
5. Have we asked the stakeholders about conscious, unconscious and undreamed of requirements? Can you show that a modeling effort has taken place to discover the unconscious requirements? Can you demonstrate that brainstorming or similar efforts taken place to find the undreamed of requirements?
6. Is every requirement in the specification relevant to this system?
7. Does the specification contain solutions posturing as requirements?
8. Is the stakeholder value defined for each requirement?
9. Is each requirement uniquely identifiable?
10. Is each requirement tagged to all parts of the system where it is used? For any change to requirements, can you identify all parts of the system where this change has an effect?

Testing Use Cases
After the requirements are tested they evolve in a functional model (for instance Use Cases) of the required application. To Test the Use Cases you can use the checklist I derived from the article: "Use Cases and Testing (Lee Copeland)"
1)Syntax Testing

• Complete:
  • Are all use case definition fields filled in? Do we really know what the words mean?
  • Are all of the steps required to implement the use case included?
  • Are all of the ways that things could go right identified and handled properly? Have all combinations been considered?
  • Are all of the ways that things could go wrong identified and handled properly? Have all combinations been considered?
• Correct:
  • Is the use case name the primary actor's goal expressed as an active verb phrase?
  • Is the use case described at the appropriate black box/white box level?
  • Are the preconditions mandatory? Can they be guaranteed by the system?
  • Does the failed end condition protect the interests of all the stakeholders?
  • Does the success end condition satisfy the interests of all the stakeholders?
  • Does the main success scenario run from the trigger to the delivery of the success end condition?
  • Is the sequence of action steps correct?
  • Is each step stated in the present tense with an active verb as a goal that moves the process forward?
  • Is it clear where and why alternate scenarios depart from the main scenario?
  • Are design decisions (GUI, Database, …) omitted from the use case?
  • Are the use case "generalization," "include," and "extend" relationships used to their fullest extent but used correctly?
• Consistent:
  • Can the system actually deliver the specified goals?

2) Domain Expert Testing

• Complete:
  • Are all actors identified? Can you identify a specific person who will play the role of each actor?
  • Is this everything that needs to be developed?
  • Are all external system trigger conditions handled?
  • Have all the words that suggest incompleteness ("some," "etc."…) been removed?
• Correct:
  • Is this what you really want? Is this all you really want? Is this more than you really want?
• Consistent:
  • When we build this system according to these use cases, will you be able to determine that we have succeeded?
  • Can the system described actually be built?

3) Traceability Testing

• Complete:
  • Do the use cases form a story that unfolds from highest to lowest levels?
  • Is there a context-setting, highest-level use case at the outermost design scope for each primary actor?
• Correct:
  • Are all the system's functional requirements reflected in the use cases?
  • Are all the information sources listed?
• Consistent:
  • Do the use cases define all the functionality within the scope of the system and nothing outside the scope?
  • Can we trace each use case back to its requirement(s)?
  • Can we trace each use case forward to its class, sequence, and/or state-transition diagrams?

Pillar 3: Gathering Data and Analyzing Trends to do Business Continuity Predictions
It is NOT possible to predict Business Continuity based on the testing process of the concerning project only. It is important to get a better foundation for the decision to implement a changed or new application. In the Rosario TAP we are doing some research for Gathering Data and Analyzing Trends to do Business Continuity Predictions from three viewpoints:

1. Fault Detection Trends
2. Change Control Trends (Changes in Requirements, Specifications, LOC’s,...)
3. Project Control Trends (Estimations, Budget, Overtime,...)

We’ve just started on this topic, so I will keep you posted in later entries.

Collaboration is a critical success factor in preventing a major disruption caused by the implementation of a new or changed application! All parties within [ALM] have to work together in creating good test coverage from the early until the last phases of the projects. I am sure that only when the Quality Levels, Learning Cycles and Metrics are in place, a good Business Continuity risk advice can be given to ensure that an organization experiences an improved "business as usual" in the event of the implementation of new and/or changed functionality.

Rob

Triggered by the article "Common Test Patterns and Reuse of Test Designs" on the MSDN Test Center, I realized that we are very good in reinventing wheels!
Last week I explained two different test teams the same solution for a test problem without referring to a Test Design Pattern at all...

So, let's make my life easier. There must be a giant collection of Test Patterns on the internet. If I find the right sites, all I have to do is referring to these Test Pattern Collections...
A few hours later I woke up:
OK, there are Test Patterns out there, but they are not easy to find.
My first tip: Don’t search for "Test Patterns", otherwise you may find some...


Better is to look for "Test Design Patterns". After some time I found some promising books, and, even better, I found some patterns.
I was hoping to find Test Design Patterns on a more abstract functional level, like "Search Engine Test Pattern", "Web Shop Test Pattern" or "Database Conversion Test Pattern". But nevertheless I am happy with the ones I found.

Because I didn't find a site with a kind of index mechanism on Test Design Patterns to refer to, I started one of my own. I spent my weekend on making two
Test Design Pattern indexes: Alphabetical and Grouped.

I'll keep on looking for more, and If I can find the time, I'll try to create some "How to Test"-articles on the more abstract test problems like testing search engines, web shops, database conversions and so on... May be these will grow to Test Design Patterns we all can use.

Rob

Analyzing and fixing a bug is, without doubt, a loss of time. And the later the bug is found, the greater are the losses. Finding bugs AEAP (As Early As Possibly) becomes more and more important, especially in the complex systems we make nowadays. So, let's join the knowledge of the Developer, the Tester and the Designer and tackle this challenge!


Of course we can train the individual team members to test their own work....but, I know from my own experience, you can be blind for your own short comings.

In the Application Life Cycle we should stimulate Developers, Testers and Designers to work together in the challenge to Find Bugs AEAP (As Early As Possibly). Join their knowledge and they will accomplish better systems in less time!
For instance, let's look into the creating of Unit Test cases. The Unit Test mechanism is great, but if you don't know anything about test coverage it's almost impossible to get the advantages you want.
Let me give an example how the Developer, the Testers and the Designers can work together....

Case
Let’s say a Developer must create a component that contains the following decisions:

if C1 and (C2 or not C3)
then
        if C5 or C4
        then
                "Result_1"
        else
                “Result_2"
        end if
else
        if C3 and C4
        then
                "Result_2"
        else
                "Result_3"
        end if
end if

We want to create Unit Test cases which will Find the Bugs AEAP....That's why we must choose for the best test coverage: Modified Condition/Decision Coverage (MC/DC)

Modified Condition/Decision Coverage
Every point of entry and exit in the program has been invoked at least once, every condition in a decision in the program has taken on all possible outcomes at least once, and each condition has been shown to affect that decision outcome independently. A condition is shown to affect a decision’s outcome independently by varying just that decision while holding fixed all other possible conditions. (wikipedia.org)

It is not easy to create the test cases for the test coverage MC/DC, especially if you want a minimum amount of test cases. Here the Tester comes into the play. With the help of TMap® test specification techniques, he/she creates the (logical) test cases. In this case the Tester delivers the Developer the following 7 (logical) MC/DC test cases.



Depending on the risks, and with the help of Equivalence classes and Boundary value analysis, the physical test cases are derived.
Now, the Developer can start building the test cases into his/her component.

So, by joining their knowledge, the Developer and Tester did create, for this phase in the life cycle, the best test coverage to Find Bugs AEAP . The effect of collaboration can even be greater if the Unit Test cases, before implementing in the code, are reviewed by the Designer.

Reviewing the test cases gives the Designer insight in the assumptions and interpretations of the Developer and Tester.

Finding Bugs during this review is the ultimate AEAP!!!

Creating these test cases manually is, even for experienced testers, pretty difficult. Within Sogeti we use a tool (I made) for this work. (The generation of the test cases of this example took me 5 minutes).

Today I read a nice article: "What's In the Box?" by Mary Altman.
This article, about speculative fiction, started with a recognizable explanation of the term "black-box theory":
"The term is typically used to describe a device of which we know or care very little about its inner workings. The entire focus is on its input/output behavior—the result rather than the reasoning."

In the article, also Sociologist Bruno Latour gives a description of the "black-box theory":
"The way scientific and technical work is made invisible by its own success. ... Thus, paradoxically, the more science and technology succeed, the more opaque and obscure they become."
....
Mary ends her article with the conclusion:
"Most simply: it doesn't matter how it works. It only matters that it works."

...and what about the "black-box theory" in the software industry?

I know that many colleague testers believe this conclusion is also valid in the software industry. So, when they organize an Acceptance Test, they rely completely on the testing target: "It only matters that it works". From the perspective of the end-user they are probably right....because the end-user is not interested in (and has no knowledge of) the inner workings. So many Acceptance Tests rely on a complete black-box strategy!

I’m not one of them!

I think the test target: "It only matters that it works" is too thin. It gives no answers for: "When it works, will it keep on working?", or "How about vendor dependency?", or "Is this software maintainable?", or "Are there any hidden features?", and so on...

I think an Acceptance Test must, besides the testing "that it works", find assurance that the inner workings are validated. I agree that it should not be tested in the Acceptance Test, but still it MATTER HOW IT WORKS!!!
So how can we find assurance that it is tested? Is it possible for the software industry to give the Acceptance Test team the assurance the inner workings are validated and correct?

On these questions I am convinced that the testing community can help the developers. I believe, much effort is already done in delivering software of good quality.
But those efforts are hidden and not tuned into an efficient and effective process. Again: I think the testers can help! When testers and developers join together in describing all the quality measures (they already perform), it will be clear if enough measures are taken and/or what additional actions are needed. Chapter 7 of TMap® Next (Development Tests) gives for this inventory a small, but effective model.

The so called "Basic Quality model" describes the quality measures from four points of view:

1. Depth of test coverage;
2. Clarity (description how and when the test coverage is reached);
3. Provision of proof (what deliverables give proof of the agreed test coverage);
4. Compliance monitoring (how does the development process perform its internal monitoring).

Together, developers and testers can fill in these four points of view.

Combined with the quality measures of the requirements and design processes, this gives the Acceptance Test team the opportunity to get a "glass-box" impression how the inner workings are validated.

The glass-box effect gives also the possibility for implementing learning cycles. Implementing the "Basic Quality model" makes it possible to investigate the origin of defects, learn from it and help each other to do it better the next time!....I know...that's a little too optimistic...

So let's do it step by step: Creating a kind of glass-box by giving the Acceptance Testers the possibility to be a virtual witness of the development process would be a great first step!

pictures: www.vuurwerkboer.nl

Nobody Wants to Deliver Crap!
In the nearly 30 years I am working in the ICT world, I've seldom met someone who deliberately made errors to annoy his contractor, manager or whoever…. Everybody I met, made his or her best effort to do their job as good as possible!
Nevertheless it is obvious that after joining the individual parts, their (and also mine) team efforts mostly aren’t good enough. Lack of Quality is a costly and sometimes even a lethal problem.

Some examples:
• A poorly programmed ground-based altitude warning system was partly responsible for the 1997 Korean Air crash in Guam that killed 228 people. ( www.cbsnews.com )

• Faulty software in anti-lock brakes forced the recall of 39,000 trucks and tractors and 6,000 school buses in 2000. ( www.spaceref.com )

• Software bugs, or errors, are so prevalent and so detrimental that they cost the U.S. economy an estimated $59.5 billion annually, or about 0.6 percent of the gross domestic product, according to a newly released study commissioned by the Department of Commerce's National Institute of Standards and Technology (NIST).( June 28, 2002) ( www.nist.gov )

• During a meeting of the Mars Exploration Program Analysis Group Meeting in Washington Dc, NASA's John McNamee, Mars Exploration Program addressed the issue of the recent failure of the Mars Global Surveyor (MGS) spacecraft. Apparently incorrect software doomed the spacecraft.(January 10, 2007) ( www.spaceref.com )

Numerous quality experts did try to solve the problem of bad quality software with a huge amount of quality methods, so I won't try to add one of my own.....No , I have an opinion about Quality in a much, much smaller scale. Let's look at the following:

I think it is strange that, one ICT-assignment parallel given to, let's say, 20 individuals, results in many different outcomes; the change that their work has the same outcome is practically NILL!
And another thing I am amazed about: If you ask those 20 individuals, whether they are certain about the quality of their work. Mostly they can’t predict the results of the connecting test or review at all. Most of the time you get answers like: "I hope I did well" or "I think they won't find too many errors".

I hope she will not yell at me
I'm glad my dentist, my general practitioner or car mechanic don't work the same way!!
I think, if you look at the way the individual, in general, is doing his work in the ICT industry, huge improvements can be made, by investing in baby steps!

Assuming that nobody wants to deliver crap: how can I help an individual (let's call him Mr. A) to do his work the right way?

I think Mr. A can be helped with 5 Simple Rules in the 3 stages of his ICT activity:
1) The assignment
2) Support and/or coaching on the job
3) Delivery and feedback

1)The Assignment

Simple Rule 1: The Assignment for the individual must be SMARTER.
SMARTER is a mnemonic used in project management at the objective setting stage. It is a way of evaluating if the objectives that are being set are appropriate for the individual project . ( wikipedia )
A SMARTER objective is one that is Specific, Measurable, Achievable, Relevant, Time-bound, Evaluated and Rewarding.

The assignment must give ALL the information that is needed to do the job the Right Way the First Time. Including the required knowledge and experience, the reference to the instructions for the techniques and tools, the acceptance criteria (a mandatory list with quality checks), and a contact for asking questions (obligatorily documented; for instance by e-mail).
Before Mr. A starts his assignment he MUST know exactly what he must do to complete the job and gets his applause. It is very important that Mr. A realizes that only he can judge if the assignment is SMARTER "enough".

Simple Rule 2: Prevent Mr. A to do the best he can! Good is good enough!
An extra note on the "Achievable"-part:

Nobody can build a Rolls Royce for the price of a 2CV
An assignment concerning Functionality, Time, Money and Quality must be in balance! It must be clear at the start what must be done if the balance is disturbed!

2) Support and / or coaching on the job

Simple Rule 3: Fight the biggest enemies of Quality: Assumptions and Interpretations.
To fight the damage of assumptions, Mr. A can be helped in two ways:
First, we must Mr. A help to make the correct interpretations, so it is very important to give a proper kick-off. If Mr. A doesn't know how the product, he is creating, will be used, he never can make the right interpretations.
Second, Mr. A must learn to ask questions when he is not exactly sure about the choices he has to make during his activity.

3) Delivery and feedback

Simple Rule 4: Delivery must be done in a way that Mr. A is forced to know the quality of his product.
Delivery can only be allowed if the product is compliant to the exit criteria. Mr. A must make a statement; he did perform the complete set of quality-checks (exit criteria).

Simple Rule 5: Feedback must be done in a way Mr. A can learn from his mistakes.
If in a later stage an error has been found, feedback must be given in two ways.
First to create the learning cycle, Mr. A must give, after analyzing the problem, an explanation why the error has slipped through his own detection.
Second, of course, the error must be corrected.

I don't know if it works for Mr. A, but I do know the 5 simple rules helped me a lot! Feel free to try them. Suggestion: try them step by step...baby steps....

I'm curious.... Can 5 Simple Rules affect the ICT Quality problem?
In other words: Can 5 Simple Rules make the ICT world stop running.... to learn baby steps?

In theory....every little bit can help...
In practice.....probably not....but I'll see...

In the mean time... I am satisfied making progress with my colleague Developers and Testers in my own surrounding...


And for the rest of the world.....don't worry.....

you've never time to do IT right...

but always time to do IT over!

Rob Kuijt