Pairwise Testing in Agile World

The fundamental challenge in testing any system is that the exhaustive testing is often impossible! This is owing to a large number of combinations of inputs to the system under test. The number grows exponentially high as the number of input parameters and the number of values each of those parameters can take, increases. Take for example, the aircraft cockpit control system which has at least 100 switches and controls.

cockpit

Even if each of these controls take 2 values (ON or OFF), it takes 2 ^ 100 = 1.2676506e+30 combinations to thoroughly test the system. That takes more than a lifetime for someone to test this system!! Not to forget the regression testing when things change. World is moving towards agile methodologies which means shorter release cycles. So quick test feedback on every build is absolutely necessary. Even with 100% automation of these test cases, it takes several days/months to give this feedback. So, what is the alternative?

The combinatorial techniques like Pairwise testing (also known as All Pairs) hold some promise. The intuition behind Pairwise technique is that some problems only occur as the result of an interaction between pairs of parameters/components. For e.g, a failure occurs if pressure < 10 & volume > 300 (2-way interaction).

The researchers Wallace and Kuhn at NIST (National Institute of Standards and Technology) have determined that 98% of the reported software defects in recalled medical devices could have been detected by testing all pairs of parameter settings.

An Example:

Consider a system that takes 3 input parameters. Each parameter takes 2 values.

example

Total no of combinations required to test this system = 2^3 = 8

all_comb

Pairwise Reductions by hand

We can get rid of test case T2, since all pairs of this combination are covered in at least one other combination other than itself. Hence it is redundant.

  • AC is covered in T1
  • CF is covered in T6
  • AF is covered in T4

With little bit of paperwork, you’ll be able to get rid of T3, T5 & T8.

reduced_subset

So the reduced subset consists of just 4 test cases. Not a great reduction but the quantum of reduction is significant when there are more no of input parameters and their values.

final_subset

Fortunately we don’t have to do this by hand. There are plenty of tools available in the market that can do this for us. Like PICT from Microsoft, TestCover etc.

Caveat

1. Pairwise testing will not be effective if you choose the wrong input test data values 

2. Pairwise testing will not be effective if you don’t have a good enough oracle. We often assume defects can be immediately observed. In fact, there may be slow internal corruption occurring

3. High-use (including defaults) or high-risk combinations probably don’t get enough attention

4. We rarely understand the n-wise connectivity within programs we are testing, so it’s not clear that pairwise testing is an appropriate choice

Note: These dangers are not exclusive to pairwise testing. They affect all testing efforts

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s