designing test

Designing Test Suites

Software Testing

Testing is an important part of the development life cycle of a software. Appropriate testing methods are necessary for ensuring the reliability of a program/software so developed. According to the ANSI/IEEE 1059 standard, testing is described as the process of analyzing a software item to detect the differences between existing and required conditions, i.e., defects/errors/bugs, and to evaluate the features of the software item. The purpose of testing is to verify and validate a software and to find the defects present in it, if any, with the subsequent aim of fixing them.

  • Verification is the process of checking a software for consistency and conformance by evaluating the results against pre-specified requirements.
  • Validation looks at the system’s correctness, i.e., the process of checking that what has been done is what the user actually wanted.
  • Defect is a variance between the expected and actual result. A defect’s source may be traced to a fault introduced in the specification, design, or development (coding) phases.

Standards for Software Test Documentation

IEEE 829-1998, known as the 829 Standard for Software Test Documentation, specifies the form of a set of documents for use in software testing [i]. There are other different standards some of which are discussed below.

  • IEEE 1008, a standard for unit testing
  • IEEE 1012, a standard for Software Verification and Validation
  • IEEE 1028, a standard for software inspections
  • IEEE 1044, a standard for the classification of software anomalies
  • IEEE 1044-1, a guide to the classification of software anomalies
  • IEEE 830, a guide for developing system requirements specifications
  • IEEE 730, a standard for software quality assurance plans
  • IEEE 1061, a standard for software quality metrics and methodology
  • IEEE 12207, a standard for software life cycle processes and life cycle data
  • BS 7925-1, a vocabulary of terms used in software testing
  • BS 7925-2, a standard for software component testing

Testing Frameworks

Testing frameworks help in writing and executing test cases easier for a given programming language or technology platform. In the following, we mention a few (amongst many) such frameworks that are commonly used.

  • jUnit – for writing unit test cases with Java [ii]
  • Selenium – is a suite of tools for automating web applications for software testing purposes. It also has a plugin for Firefox [iii]
  • HP QC – is the HP Web-based test management tool. It familiarizes with the process of defining releases, specifying requirements, planning tests, executing tests, tracking defects, alerting on changes, and analyzing results. It also shows how to customize project [iv]
  • IBM Rational – Rational software has a solution to support business sector for designing, implementing, and testing software [v]

Need for Software Testing

There are many reasons for why we should test software, such as:

  • Software testing identifies any existing faults in a software. Removal of such faults helps to reduce the number of system failures, which improves the reliability and the quality of the system.
  • Software testing can also improve other system qualities, such as maintainability, usability, and testability.
  • In order to meet the different legal requirements.
  • In order to meet industry specific standards, such as the Aerospace, Missile and Railway Signaling standards.

Test Cases and Test Suite

A test case describes an input and its corresponding expected output. Inputs are of two types: preconditions (circumstances that hold prior to the test case execution) and the actual inputs that are identified by some testing methods. A set of test cases is called a test suite.

Types of Software Testing

Testing is done at every stage of software development life cycle, but with different objectives at different levels. There are different types of testing, such as stress testing, volume testing, configuration testing, compatibility testing, recovery testing, maintenance testing, documentation testing, and usability testing. In the following, we briefly discuss a few categories of testing [1]

Unit Testing

Unit testing is done at the lowest level. It tests the basic unit — the smallest testable piece — of software. Unit testing are of two types.

  • Black box testing: This is also known asfunctional testing, where the test cases are designed based on input-output values only. There are different types of Black Box Testing, such as:

    Equivalence class partitioning: In this approach, the domain of input values to a program is divided into a set of equivalence classes. E.g., consider a software program that computes whether an integer in the range of 0 to 10 is even or not. There are three equivalence classes for this program: 1) The set of negative integers, 2) Integers in the range 0 to 10, and 3) Integers larger than 10

    Boundary value analysis: In this approach, the values at boundaries of different equivalence classes are taken into consideration. E.g., in the above given example, a boundary values-based test suite is { 0, -1, 10, 11 }

  • White box testing: It is also known as structural testing. In this methodology, test cases are designed on the basis of examination of the code. This testing is performed based on the knowledge of how the system is implemented. It includes analyzing data flow, control flow, information flow, coding practices, and exception and error handling within the system to test the intended and unintended software behavior. White box testing can be performed to validate whether code implementation follows intended design to validate implemented security functionality and to uncover exploitable vulnerabilities. This testing requires access to the source code. Though white box testing can be performed any time in the life cycle after the code is developed, it is a good practice to perform white box testing during the unit testing phase.

Integration Testing

Integration testing is performed when two or more tested units are combined into a larger structure. The main objective of this testing is to check whether the different modules of a program interface with each other properly or not. This testing is mainly of two types:

  • Top-down approach
  • Bottom-up approach

In the bottom-up approach, each subsystem is tested separately and then the full system is tested. Top-down integration testing starts with the main routine and one or two subordinate routines in the system. After the top-level “skeleton” has been tested, the immediate subroutines of the “skeleton” are combined with it and tested.

System Testing

System testing tends to affirm the end-to-end quality of the entire system. System testing is often based on the functional/requirement specifications of the system. Non-functional quality attributes, such as reliability, security, and maintainability are also checked. There are three types of system testing

  • Alpha testing is done by the developers who develop the software. This testing is also done by the client or an outsider with the presence of developer or tester.
  • Beta testingis done by very few number of end-users before the delivery, where the change requests are fixed, if the user gives any feedback or reports any type of defect.
  • User Acceptance testing is also another level of the system testing process where the system is tested for acceptability. This test evaluates the system’s compliance with the client requirements and assess whether it is acceptable for software delivery.

An error correction may introduce new errors. Therefore, after every round of error-fixing, another testing is carried out called regression testing. Regression testing does not belong to either unit testing, integration testing, or system testing, instead, it is a separate dimension to these three forms of testing.

Regression Testing

The purpose of regression testing is to ensure that bug fixes and new functionality introduced in a software do not adversely affect the unmodified parts of the program [2]. Regression testing is an important activity at both testing and maintenance phases. When a piece of software is modified, it is necessary to ensure that the quality of the software is preserved. To this end, regression testing is to retest the software using the test cases selected from the original test suite.


Write a program to calculate the square of a number in the range 1-100

#include <stdio.h>
int main()  {
     int n, res;
     printf("Enter a number: ");
     scanf("%d", &n);
     if (n >= 1 && n <= 100)
         res = n * n;
          printf("\n Square of %d is %d\n", n, res);
     else if (n <= 0 || n > 100)
         printf("Beyond the range");

      return 0;


Inputs               Outputs
I1 :  -2        O1 :  Beyond the range
I2 :   0        O2 :  Beyond the range
I3 :   1        O3 :  Square of 1 is 1
I4 : 100        O4 :  Square of 100 is 10000
I5 : 101        O5 :  Beyond the range
I6 :   4        O6 :  Square of 4 is 16
I7 :  62        O7 :  Square of 62 is 3844

Test Cases

T1 : {I1, O1}
T2 : {I2, O2}
T3 : {I3, O3}
T4 : {I4, O4}
T5 : {I5, O5}
T6 : {I6, O6}
T7 : {I7, O7}

Some Remarks

Beginners often have a misconception that one should be concerned with testing only after coding ends. Testing is, however, not a just phase towards the end. It is rather a continuous process. The efforts for testing should begin in the form of preparation of test cases after the requirements have been finalized. The Software Requirements Specification (SRS) document captures all features to be expected from the system. The requirements so identified there should serve as a basis towards preparation of the test cases. Test cases should be designed in such a way that all target features can be verified. However, testing a software is not only about proving that it works correctly. Successful testing should also point out the bugs present in the system, if any.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s