Link Search Menu Expand Document

Lab 5: Mutation testing

Purpose

The purpose of this lab assignment is to present a contemporary tool and techniques to conduct mutation testing on real software, in a way that it is often done in the Industry.

Introduction

Mutation testing is a software testing technique and an active research area. It was first proposed in 1979. The purpose of mutation testing in simple words is, “Does our test really test what we think that they test?”.

There are numerous tools in active development to conduct mutation testing. For Java which has been the primary language in the labs of this course so far, an important tool is PIT. However, in this lab, we will introduce Dextool. Dextool is an open-source tool proposed by SAAB Aeronautics to develop and maintain high-quality tests.

Preparation

The lab will involve writing a few lines of C++. It is not required that you understand all the caveats of C and C++. However, we encourage you to get acquainted with the documentation for Google test.

Dextool

Dextool Mutate Dextool is a framework for writing plugins using libclang. The main focus is on tools for testing and static analysis. In a standard installation of Dextool, six different plugins are included. The mutation testing plugin, Dextool Mutate, functions in such a way that the user provides a configuration-file where scripts, settings, and different paths are specified. The picture above shows the flow for the plugin, where the test part of mutation testing, is depicted in a more detailed manner. As shown in the image, the plugin is divided into different parts (executable commandos) - analyze, test and report. There are two more executable commandos for Dextool Mutate, admin and generate, but these can be ignored in this lab. See Dextool Mutate README for more information.

Setup

NOTE If you use thinlinc follow the tutorial below:

Using dextool on thinlinc

If you wish to use dextool on thinlinc there exists binaries that allows you to do so. open the thinlinc terminal and run the following commands:

module add courses/TDDD04
alias dextool=/courses/TDDD04/dextool/dextool

In the given terminal execute:

dextool --help

The expected output should be:

usage:
 dextool <command> [options] [<args>...]

options:
 -h, --help         show this global help
 -d, --debug        turn on debug output for detailed tracing
 --version          print the version of dextool
 --plugin-list      print a list of plugins

commands:
  help
  uml           generate PlantUML and GraphViz diagrams
  graphml       static code analyse information as GraphML
  mutate        mutation testing plugin
  ctestdouble   generate a C test double. Language is set to C
  cpptestdouble generate a C++ test double. Language is set to C++
  fuzzer        generate a wrapper for a block of code to fuzz with AFL
  analyze       static code analysis of c/c++ source code
  example       print all AST nodes of some c/c++ source code


See 'dextool <command> -h' to read about a specific subcommand.

Now let’s see if we have all the plugins for dextool. Run the following command:

dextool --plugin-list

The output should be:

uml
graphml
mutate
ctestdouble
cpptestdouble
fuzzer
analyze
example
  • Clone the repository by executing the following in your command line terminal:

      git clone  https://github.com/joakim-brannstrom/dextool.git --branch v1.3.2 --single-branch
    

This is the repository that you will use for the subsequent thinlinc instructions

NOTE: You are not supposed to run the steps in course/TDDD04/dextool but rather in your local copy of dextool

Using the latest version of Dextool (If you wish to conduct the lab on your own machine)

In order to set up and use Dextool, it needs to build first. See the README for a more detailed walkthrough if you wish to use the latest version of Dextool.

The steps should provide you with a freshly built version of Dextool and also the executable binaries for the different plugins in a standard installation. (don’t hesitate to create issues on the Dextool github-page if anything needs to be improved or clarified in the README-guide).

NOTE: This option is might be harder to setup and install compared to the thinlinc alternative.

Part 1 - Manual mutation testing

Consider the following code snippet written in C:

//Simple fizzbuzz may or may not be correct
#include <stdio.h>

void fizz_buzz()
{
    for (int i = 1; i < 35; ++i) {
    //% is the modulo operator in C.
        if (i % 3 == 0 && i % 5 == 0) {
            printf("Fizz!Buzz!\n");
        } else if (i % 3 == 0) {
            printf("Fizz\n");
        } else if (i % 5 == 0) {
            printf("Buzz\n");
        } else {
            printf("%d\n", i);
        }
    }
}
void main()
{
    fizz_buzz();
}

Select five different values of the loop integer i. For these values, we would like you to give us the expected output.

Present the expected output for a specific value of i to us

We would like you to introduce five mutants of your choice. Describe how that would affect the result for the five different values of i you selected. Present the findings of this small investigation and in your report, include a table reporting these results.

For this small exercise we would like that you apart from providing a structured answer to your investigation provide us with an answer to the following questions:

  • What is the practical use case of introducing mutants. Say that the five pairs of input and output printed to the terminal was our tests?

  • Why would we in an industrial context want to automate the process of introducing mutants?

Part 2 - Automatic mutation testing

First of all, make sure that you have done the Preparation. In this section, we would like you to use mutation testing on a small roguelike game. Instructions described and presented in the READMEs assumes that you are in the correct directory before executing any commands/scripts, plugin/mutate/examples/game_tutorial/.

Running on your own machine via a custom installation

Read the instructions for game tutorial.

Running on thinlinc

Use the repository you previously cloned.

If you forgot to clone it execute the following command:

    git clone  https://github.com/joakim-brannstrom/dextool.git --branch v1.3.2 --single-branch

Read the instructions for game tutorial.

There exist a few tests for this game already. After you have read the instructions, we would like you to run dextool in the way that it is described in the README. Once you have done so, you should analyze the results.

NOTE: that the game tutorial is located in dextool/plugin/mutate/examples/game_tutorial

  • What do the results of running dextool tell us about test quality? Relate the findings from doing this experiment to the first lab. Would we make similar observations for tests for highly coupled code as in the Colony example?

Hint: Consider the test case similarities!

Part 3 - Extend the test suite

In this part, we would like you to extend the test suite. Provide extensions to the test suite with the goal of improving the mutation score and optionally coverage. For each new test, you add to provide a rationale for why this particular new test is being introduced. For passing (Grade 3), it is enough that the mutation score is somewhat improved.

Hint: Use the results from the previous part, i.e., the generated mutation and coverage reports.

Hint: In the game tutorial directory one can also run to build ./build.sh and ./test.sh to run googletest.

Hint: If the testsuite is flaky dextool will print the following message error: Test suite is unreliable. It must return exit status '0' when running with unmodified mutants this means that your testsuite is unreliable. Test it by running ./test.sh a test might be failing and you can see it quickly via that command.

Attach all the tests you introduced to your report along with your rationale

Part 4 - Equivalent mutants

Equivalent mutants and how to deal with these is an active research topic. We would like you to read up on what an Equivalent mutant is and then do the following tasks:

  • Find at least one equivalent mutant in the fizz_buzz example. You might have already found one!

  • If we consider mutation score when doing automated mutation testing. Why is the existence of equivalent mutants a practical problem?

Reporting your results

See the general instructions for successfully completing the lab assignment. Write a lab report that answers the questions stated above.

For Part 2 and Part 3 we would like to see screenshots of the tool in action that motivate the answers to the questions raised in those sections.