Mutation testing: set up back-end projects with StackSpot EDP

StackSpot EDP is a robust platform for test automation, making it easy to create and extend automation projects. For mutation testing, StackSpot EDP excels as an efficient solution, enabling easy replication of configurations across multiple projects. 

Thanks to the platform’s flexibility, few modifications are needed, regardless of the technology used, speeding up the adoption of mutation tests in diverse development scenarios.

This article explains how to set up mutation tests in back-end projects using a Plugin developed with StackSpot EDP.

What are mutation tests?

Mutation testing is a strategy used to assess the quality of an application’s software tests.

Imagine a program that executes various functions with tests to ensure that everything runs as expected. Mutation testing verifies that these tests are effective in confirming the software works properly.

How mutation testing works

Mutation testing basically involves introducing minor errors, called mutants, into the program code to check if the tests can detect these changes. 

The ultimate goal is for robust tests to fail when they find these intentional changes. If the tests are unable to detect the mutants, it suggests they are ineffective at identifying problems. 

This technique employs a failure-based testing strategy to evaluate and enhance test quality.

Mutation testing process

The mutation testing process involves the following steps:

• Introducing faults: Changes are made to the program’s source code to create mutants, such as replacing a plus sign with a minus sign in a mathematical expression.
• Test application: Test cases are applied to both the original code and the mutated program. If test results for the original code and the mutants differ, the mutant has been detected and eliminated, demonstrating the test’s effectiveness.
• Analysis of results: If the tests fail to detect the change and the results for the original code and the mutant are the same, the tests lack sensitivity to identify errors, indicating a need to improve the test cases.

Examples of mutation tests

Let’s consider the following example using Java:

Original code:

public class Calculator {
    public int sum(int a, int b) {
        return a + b;
    }
}

Mutant:

public class Calculator {
    public int sum(int a, int b) {
        return a - b; // Intentional alteration
    }
}

Unit Testing:

import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.assertEquals;

public class CalculatorTest {
    @Test
    public void testSum() {
        Calculator calc = new Calculator();
        assertEquals(5, calc.sum(2, 3)); // Test that covers the original code
    }
}

In the example above, the unit test testSum checks whether the sum method of the Calculator class returns 5 for the values 2 and 3. In the original code, the sum of 2 and 3 is 5, and the test passes correctly. 

However, when the code is changed to a – b (mutating code), the result of the sum of 2 and 3 is -1, but the test still passes because it doesn’t detect the intentional change in the code.

Changes to a mutated program

Several techniques can be used to generate mutants, with the most common being:

• Operand substitution operators that replace an operand with another or constant value. For instance, a number in an expression may be replaced with a fixed value.
• Expression modification operators that alter the original expression by inserting, replacing or removing operators. For example, an addition operator could be changed to a subtraction.
• Statement modification operators that modify the original statement by adding or removing existing statements, potentially altering the logic of a block of code.

These techniques can be combined to create mutants with different types of faults, increasing the effectiveness of the mutation test.

Benefits of implementing mutation testing

Incorporating mutation testing into your software project offers multiple benefits:

• Improves test quality: By ensuring tests can catch mutating code, you verify that they can detect subtle errors in the code.
• Boosts software robustness: More effective tests ensure that the software will behave correctly, even after modifications or updates to the code.
• Reduces the risk of failures: By identifying weaknesses in the tests, you can improve them, reducing the risk of undetected errors affecting the software’s performance.

While mutation testing may seem complex, it is a powerful tool for improving the quality and reliability of software testing, ensuring that your code works as expected under various conditions.

Tools for mutation testing

Several tools are available for mutation testing in Java or Kotlin projects. One of the best-known is Pitest, which will be used in this article.

For more information on mutant generation approaches, see the Pitest tool documentation.

Pre-requisites

• Java JDK.
• StackSpot CLI. Consult the documentation for installing the StackSpot CLI.

Example project

This article will explore the Pequeno-Investidor project, a personal application developed in Java that delivers financial market information through an API. The API gathers data through web scraping, ensuring the information is both current and relevant.

The project was structured using the Maven dependency manager. The project’s structure is organized as follows:

├───src
│   ├───main
│   │   ├───java
│   │   │   └───com
│   │   │       └───money
│   │   │           └───pequenoinvestidor
│   │   │               ├───configuration
│   │   │               ├───controller
│   │   │               ├───model
│   │   │               ├───services
│   │   │               │   └───imp
│   │   │               └───util
│   │   └───resources
│   └───test
│       └───java
│           └───com
│               └───money
│                   └───pequenoinvestidor
│                       └───integração

Getting to know the StackSpot EDP Plugin

In this tutorial, we’ve created a Plugin using StackSpot EDP for demonstration purposes on a personal GitHub account. These accounts enable users to develop solutions and experiment with StackSpot EDP. The following code served as the foundation for creating the Plugin: plugins-stackspot-edp.

The solution was published following the instructions in the official StackSpot EDP documentation. First, a “qa-stack” was created in an associated personal account (learn more details here). Next, the “mutation-java” Plugin was published (step-by-step instructions here) to simplify the configurations for running mutation tests on Java projects.

Additional details on the content creation process are available in the official StackSpot EDP documentation.

Using the Plugin

To use the mutation-java Plugin, ensure you are in the directory of the desired application. In this example, we are using the Pequeno Investidor application.

cd pequenoinvestidor

In the root of the project, run the following command to apply the Mutation Test Configuration Plugin to your project:

stk apply plugin qa-tools/mutation-java@0.1.1

You will need to supply some information to configure the Plugin. The first piece required is the dependency manager used in the project. The Plugin supports both Maven and Gradle projects (using Groovy). In this example, we’ll use Maven.

Which dependency manager for the project?

 » 1) Maven

   2) Gradle

  Answer: 1) Maven

The next piece of information required is the logging library used in the project. In the Pequeno Investidor project, we are using the org.slf4j.Logger.

Which logging lib is used in the project? (Use shortcuts or arrow keys)

   1) java.util.logging

   2) org.apache.log4j

 » 3) org.slf4j

   4) org.apache.commons.logging

  Answer: 3) org.slf4j

Note: Log snippets can interfere with the results of mutation tests. To avoid this, configure Pitest to ignore log calls by using avoidCallsTo.

Define the test coverage

Next, the expected mutant coverage for the project will be defined after running the mutation tests. Given that the “Small Investor” project has low unit test coverage, we have set a target of 50% for mutant coverage.

 What mutant coverage is expected?(1% a 99%) 

Note: Mutant coverage is calculated as the percentage of dead mutants relative to the total number of mutants generated: (Dead Mutants / Total Number of Mutants) × 100. 

This indicator is essential for evaluating the effectiveness of unit tests in identifying mutants. Therefore, a high rate of surviving mutants indicates low mutant coverage, suggesting that the tests may not be detecting all possible defects.

Configure the group of mutants 

Next, you must specify the mutant group configured for your project. This configuration defines the number and types of mutants that will be generated. By default, the PITest tool uses OLD_DEFAULTS mutants. However, in this configuration, we have opted for the DEFAULTS option.

Group of mutants to be used?
   1) OLD_DEFAULTS
 » 2) DEFAULTS
   3) STRONGER
   4) ALL
  Answer: 2) DEFAULTS

Note: More details about the mutations generated in the application code based on the chosen group can be found in mutators.

We now have two critical pieces of information. First, we need to inform Pitest of the package where the application’s business classes are to ensure it generates the mutants correctly. 

While it is possible to use a generic pattern like com.* to generate mutators for all the classes in the project, this approach is not advisable. In this example, we’ll specify the package com.money.pequenoinvestidor.services.imp.CalculoServiceImp.

Which package of classes to generate mutantes?com.money.pequenoinvestidor.services.imp.CalculoServiceImp

Note: In mutation testing, it is crucial to focus on the classes where the mutations will be applied. The goal is to ensure that the unit tests can detect changes to the business rules. 

Depending on the project’s architecture, mutations should be focused on the classes that implement these rules, while ignoring interfaces, abstract classes, and other elements that could skew the mutation test results.

Specify the package

We need to specify the package for our project’s test classes. In this example, we’ll use the package com.money.pequenoinvestidor.TestCalculoServiceImp.

What is the unit test package? com.money.pequenoinvestidor.TestCalculoServiceImp

Once the process is complete, StackSpot will confirm that the Plugin has been successfully applied to the project.

- Plugin qa-tools/mutation-java@0.1.1 aplicado.

By accessing the pom.xml file of the Small Investor project, we can see that the Pitest Plugin settings were applied correctly, with the provided inputs serving as the foundation for configuring the mutation tests.

<plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.0.0-M7</version>
                <configuration>
                    <excludes>
                        <exclude>**/FiisTest.java</exclude>
                    </excludes>
                </configuration>
                <dependencies>
                    <dependency>
                        <groupId>org.junit.jupiter</groupId>
                        <artifactId>junit-jupiter-engine</artifactId>
                        <version>5.4.0</version>
                    </dependency>
                </dependencies>
            </plugin>
            <plugin>
                <groupId>org.pitest</groupId>
                <artifactId>pitest-maven</artifactId>
                <version>1.5.0</version>
                <configuration>
                    <targetClasses>
                        <param>com.money.pequenoinvestidor.services.imp.CalculoServiceImp</param>
                    </targetClasses>
                    <targetTests>
                        <param>com.money.pequenoinvestidor.TestCalculoServiceImp</param>
                    </targetTests>
                    <mutators>
                        <mutator>DEFAULTS</mutator>
                    </mutators>
                    <outputFormats>
                        <outputFormat>HTML</outputFormat>
                        <outputFormat>XML</outputFormat>
                    </outputFormats>
                    <failWhenNoMutations>false</failWhenNoMutations>
                    <avoidCallsTo>
                        <avoidCallsTo>org.slf4j</avoidCallsTo>
                    </avoidCallsTo>
                    <mutationThreshold>50</mutationThreshold>
                </configuration>
            </plugin>
        </plugins>

Running the mutation tests

With the Pitest tool configured in the project, mutation tests can be run quickly by executing the following command in Maven:

mvn test-compile org.pitest:pitest-maven:mutationCoverage

We’ll have a similar exit:

--------------------------------------------------------------------------------
> Total  : 1 seconds
--------------------------------------------------------------------------------
================================================================================
- Statistics
================================================================================
>> Generated 14 mutations Killed 11 (79%)
>> Ran 19 tests (1.36 tests per mutation)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  2.977 s
[INFO] Finished at: 2024-07-12T18:28:29-03:00
[INFO] ------------------------------------------------------------------------

Note: Although we set a coverage limit of 50%, the results for the CalculoServiceImp class were promising: 79% of the 14 mutants generated were detected by the existing unit tests.

Pitest automatically generates output reports, allowing you to identify key details, such as coverage data:

Check out the types of mutants generated based on the selected group, along with specific details for each mutation:

Conclusion

Writing unit tests is just one step in the process; ensuring their quality is essential. Mutation testing is a powerful tool for uncovering flaws in code logic.

This is another practical example of how StackSpot EDP facilitates test application and creation, improving software quality. The tool simplifies the configuration of mutation tests, making the process more efficient and easy to replicate. 

Do you have any questions or suggestions? Feel free to leave a comment.