Link Search Menu Expand Document
Software testing (TDDD04)

Lab 3: Mutation Testing (Updated for 2023)

Purpose

The purpose of this lab assignment is to present a contemporary tool and techniques to conduct mutation testing on real software, in a way that it is often done in the Industry.

Introduction

Mutation testing is a software testing technique and an active research area. It was first proposed in 1978. The purpose of mutation testing, in simple words, is, “Does our tests test what we think they test?”.

There are numerous tools in active development to conduct mutation testing. For Java an important tool is PIT. However, in this lab, we will introduce Dextool. Dextool is an open-source tool proposed by SAAB Aeronautics to develop and maintain high-quality tests.

Preparation

The lab will involve writing a few lines of C++. It is not required that you understand all the caveats of C and C++. However, we encourage you to get acquainted with the documentation for Google test.

Dextool

Dextool Mutate Dextool is a framework for writing plugins using libclang. The main focus is on tools for testing and static analysis. In a standard installation of Dextool, six different plugins are included. The mutation testing plugin, Dextool Mutate, functions in such a way that the user provides a configuration file where scripts, settings, and different paths are specified. The picture above shows the flow for the plugin, where the test part of mutation testing is depicted in a more detailed manner. As shown in the image, the plugin is divided into different parts (executable commandos) - analyze, test, and report. There are two more executable commands for Dextool Mutate, admin and generate, but you can ignore these in this lab. See Dextool Mutate README for more information.

Setup

NOTE If you use thinlinc/the IDA Lab Computers follow the tutorial below:

Using dextool on the IDA Lab Computers

In this lab, we recommend that you work on the IDA Lab Computers using the provided apptainer container. The container comes prepackages with the software to run the lab; no extra configuration should be required.

Note During the lab dextool related commands must be executed in the Apptainer shell.

  1. Start by adding the course module and Apptainer:

     module add courses/TDDD04
  2. Copy the apptainer container from the course folder and add it to a folder of your choosing: 
     cp /courses/TDDD04/dextool-image/dextool.sif <folder-you-want-the-dextool-image-in>

  3. Run Apptainer by executing:

     apptainer run dextool.sif

    This should result in a new shell spawning:

     Apptainer>
  4. In this shell, execute: 
     Apptainer> dextool --help

The expected output should be:

usage:
 dextool <command> [options] [<args>...]

options:
 -h, --help         show this global help
 -d, --debug        turn on debug output for detailed tracing
 --version          print the version of dextool
 --plugin-list      print a list of plugins

commands:
  help
  analyze       static code analysis of c/c++ source code
  cpptestdouble generate a C++ test double. Language is set to C++
  ctestdouble   generate a C test double. Language is set to C
  mutate        mutation testing plugin
  uml           generate PlantUML and GraphViz diagrams

See 'dextool <command> -h' to read about a specific subcommand.

  1. Now, let’s see if we have all the plugins for dextool. Run the following command:

     Apptainer> dextool --plugin-list

The output should be:

    Apptainer> dextool --plugin-list
        analyze
        cpptestdouble
        ctestdouble
        mutate
        uml

  1. Clone the dextool repository into a suitable directory by executing the command in your command line terminal:

     git clone https://github.com/joakim-brannstrom/dextool.git --branch v5.0.0 --single-branch

You will use this repository in part 2 of this lab, in which you will work in:

<The directory you cloned dextool in>/dextool/plugin/mutate/examples/game_tutorial

Using the latest version of Dextool (If you wish to conduct the lab on your machine)

NOTE: A custom installation might be harder compared to using the apptainer image and the assistants might be unable to assist you with this setup.

  • To set up and use Dextool, it needs to build first. See the README for a more detailed walkthrough if you wish to use the latest version of Dextool.

  • The steps should provide you with a freshly built version of Dextool and the executable binaries for the different plugins in a standard installation. (don’t hesitate to create issues on the Dextool github-page if anything needs to be improved or clarified in the README-guide).

  • As an alternative it is also possible to use the Apptainer image on your own machine. In this case you need to download the dextool.sif file from your IDA System and use it on your own laptop with your own installation of Apptainer.

Troubleshooting Apptainer (2023)

There are some issues with apptainer and symbolic links on the IDA-Systems that have been observed on some user accounts. In principle it boils down to issues with the file system, and similar issues have also been discovered in TDDE45.

If you encounter input/output errors or similar it is possible to resolve it by deleting the apptainer cache. This can be done by running:

$ rm -rf   ~/.apptainer/cache

Once that is done the local cache of apptainer should be deleted. Afterwards you can restart Apptainer and proceed in the lab.

!!Note, be very careful when copying this command!!

An Alternative to Cache Deletion

As an alternative to the above, if the error is presistent it is possible to conduct the Lab in the /var/tmp folder. This should work without having to reset the cache from time to time (It has to do with how the filesystem works), still, it should be noted that the data in this folder may get deleted during system updates, therefore it is recommended that you backup your work by copying it to the desktop with some regularity besides the usual version handling you have been using in the course.

See the instructions under header Alternative to running the Lab to avoid some of the issues on the lab system (2023)

Part 1 - Manual mutation testing - Mandatory

In this part of the lab, we will do mutation testing without using any auxiliary tools. Consider the following program written in C:

//Simple fizzbuzz may or may not be correct
#include <stdio.h>

void fizz_buzz()
{
    for (int i = 1; i < 35; ++i) {
    //% is the modulo operator in C.
        if (i % 3 == 0 && i % 5 == 0) {
            printf("Fizz!Buzz!\n");
        } else if (i % 3 == 0) {
            printf("Fizz\n");
        } else if (i % 5 == 0) {
            printf("Buzz\n");
        } else {
            printf("%d\n", i);
        }
    }
}
void main()
{
    fizz_buzz();
}

Write this program to a file named fizzbuzz.c, and compile it:

gcc fizzbuzz.c -o fizzbuzz

Execute the program by:

./fizzbuzz

Given that you are in the correct folder.

Select 5 values of the variable i in the for loop.

Present the expected output that the program prints for a specific value of i We would like you to introduce five mutants of your choice. Describe how that would affect the output of the program for the five different values of i you selected. Present your investigation findings in your report, and include a table reporting these results.

Alongside your previous answers, you should provide answers to the following questions:

  • What is the practical use case of introducing mutants? Say that the five pairs of input and output printed to the terminal were our tests?

  • Why would we want to automate the process of introducing mutants in an industrial context?

Part 2 - Automatic mutation testing - Mandatory

First, make sure you have done the Preparations. In this part of the lab, we would like you to use mutation testing on a small roguelike game. Instructions described and presented in the READMEs assume that you are in the correct directory (plugin/mutate/examples/game_tutorial/) and are running in the Apptainer shell before executing any commands/scripts.

Running on your machine via a custom installation

Read the instructions for game tutorial.

Running the IDA Lab Computers

Use the repository you previously cloned.

If you forgot to clone it, execute the following command:

    git clone https://github.com/joakim-brannstrom/dextool.git --branch v5.0.0 --single-branch

Read the instructions for game tutorial.

There exist a few tests for this game already. After you have read the instructions, we would like you to run dextool as described in the README.

Quickstart

There are 3 basic commands to get started.

Apptainer> ./setup.sh
Apptainer> dextool mutate analyze
Apptainer> dextool mutate test

Finally to generate the report run:

Apptainer> dextool mutate report --style html --section tc_similarity --section tc_min_set --section tc_full_overlap_with_mutation_id --section tc_killed_no_mutants --section tc_full_overlap --section trend

Note that the above command might render with linebreaks on some devices

To open the report run the following command in a separate terminal

$ firefox <your-folder-path>/game_tutorial/html/index.html &

The expected output of a succesful test run should look something like

overviewpicture

NOTE: The game tutorial is located in dextool/plugin/mutate/examples/game_tutorial

NOTE: To view the report open the generated index.html file with the browser available on the IDA-Systems

For this part provide an answer to the following question and add it to the Lab Report.

  • What do the results of running dextool tell us about the current quality of the testsuite?

Part 3 - Extend the test suite - Mandatory

In this part, we would like you to extend the test suite. Provide extensions to the test suite to improve the mutation score and, optionally, coverage.

For each new test, you add to provide a written rationale explaining why this particular new test is being introduced.

  • Attach all the tests you introduced to your report along with your rationale

  • For passing (Grade 3), it is enough that the mutation score improves relative to your baseline. However, the added test(s) should be motivated in the lab report.

Hint: Use the results from the previous part, i.e., the generated mutation and coverage reports to get an idea of where to start.

Hint: In the game tutorial directory, one can also run: cd build && make -j $(nproc) rl_test && cd .. && ./build/rl_test to run the tests with GoogleTest.

Hint: If the testsuite is flaky, dextool will print the following message error: Test suite is unreliable. It must return exit status '0' when running with unmodified mutants this means that your testsuite is unreliable.

Hint: Feel free to modify the existing program as long as you do not break any existing functionality. If you make such changes motivate them in your lab report.

Hint: It might be easier to start by testing the different functions in util.

Part 4 - Equivalent mutants - Mandatory

Equivalent mutants and how to deal with these is an active research topic. We would like you to read up on what an Equivalent mutant is and then do the following tasks:

  • Find at least one equivalent mutant in the fizz_buzz example. You might have already found one!

  • If we consider mutation score when doing automated mutation testing. Why could the existence of equivalent mutants be a practical problem?

Part 5 - Optional, for higher grades

  • Besides fulfilling the requirements for passing the lab, at least a third of the surviving mutants relative to the initial state should be eliminated by the tests introduced in Part 3. Just as in the mandatory part the new tests should be motivated and included in the final report.

Note that this number is relative your baseline, e.g how many mutants you started out with initially. In the example baseline this ammounts to 11 additional mutants

Furthermore, you should provide a written essay of a minimum of 500 words that answers the following questions:

  1. What is Mutation Testing?
  2. How can Mutation Testing improve the quality of testsuites?
  3. What are the current challenges with Mutation Testing?

This report should be written using your own words, and any claims made should be referenced using peer-reviewed sources.

Reporting your results

See the instructions on the course page general submission instructions.

Write a lab report that answers the questions stated above.

For Part 2 and Part 3 provide screenshots of the tool in action that motivate the answers to the questions raised in those sections, along with written answers to the questions for Part 1-4.

If you do the optional part of this lab also attach your findings when conducting Part 5.

Alternative to running the Lab to avoid some of the issues on the lab system (2023)

As an alternative to the issues described in the troubleshooting section. Follow the instructions as described in setup.

However, instead of running the command in the Apptainer shell, it is possible to use the exec command of apptainer.

To do this download the apptainer container as previously. However, instead of running it in the shell set an alias for dextool.sif file.

In this way it is possible to avoid the file systems

dextoolSifPath=<path-to-the-directory-where-you-downloaded-the-sif-file>/dextool.sif

Clone the repository and navigate to game_tutorial Then execute all commands in the following way:

apptainer exec $dextoolSifPath ./setup.sh

Run the analyze routine as follows

apptainer exec $dextoolSifPath dextool mutate analyze

Run the testing routine as:

apptainer exec $dextoolSifPath dextool mutate test

To generate the report execute

apptainer exec $dextoolSifPath dextool mutate report --style html --section tc_similarity --section tc_min_set --section tc_full_overlap_with_mutation_id --section tc_killed_no_mutants --section tc_full_overlap --section trend

Note, that if you by mistake run some command outside of the container you might need to rerun setup analyze and so on. Hence, for instance if you run the test command listed prior you need to prefix it with apptainer exec $dextoolSifPath <...> as well.

If you want to run the test command detailed higher up in this report, I recomend that you create a file called test.sh Then if you want to run the test you can do so by executing.

apptainer exec $dextoolSifPath ./test.sh