SuMo

SuMo is a mutation testing tool for Solidity Smart Contracts.

SuMo was designed to run mutation testing on Solidity projects in a NodeJS environment. It can run test using Hardhat, Brownie and Forge, hybrid test suites, and custom test scripts.

Table of Contents

• Installation
• Configuration
• CLI Usage
• Mutation Operators
• Publications

Installation

To install sumo run npm install @morenabarboni/sumo

Configuration ⚙️

Before using SuMo you must specify your desired configuration in a sumo-config.js in the root directory of your project. The sumo-config.js is automatically generated when SuMo is installed.

Here's a simple example of sumo-config.js:

module.exports = {  
      buildDir: "auto",                               //build directory of the SUT (auto detect)
      contractsDir: "auto",                           //contract directory of the SUT (auto detect)
      testDir: "auto",                                //test directory of the SUT (auto detect)
      skipContracts: ["interfaces", "mock", "test"],  // Relative paths from contractsDir
      skipTests: [],                                  // Relative paths from testsDir
      testingFramework: "auto",                      //testing framework (auto detect)
      minimalOperators: false,                       // use minimal mutation rules
      randomSampling: false,                         //use random mutant sampling
      randomMutants: 100,                            //if random sampling is enabled, generate 100 mutants max
      testingTimeOutInSec: 500                       //testing time-out for a mutant
}

1) SUT directories

SuMo will try to automatically find your project directories based on standard naming conventions (e.g., /contracts and /test). These can be overriden in the sumo-config.js file. Contracts and test files/folders to be ignored by SuMo can be specified as well.

2) Testing Frameworks Configuration 🔗

These fields allow to customize the testing frameworks used by SuMo.

By default, testingFramework is set to auto: SuMo will automatically select the testing framework(s) to be used based on the configuration files (e.g., foundry.toml) present in your workspace. If multiple configuration files are found, SuMo will try to run a hybrid testing process.

Brownie

When choosing brownie:

• SuMo will rely on a local/global brownie installation;
• The smart contracts will be compiled with a minimal compile command (e.g., brownie compile );
• The smart contracts will be tested with a minimal test command and (optionally) by a list of test files to be executed (e.g., brownie test ...testFiles --exitfirst) .

Forge

When choosing forge :

• SuMo will rely on the global installation of foundry;
• The smart contracts will be compiled with a minimal compile command (e.g., forge build);
• The smart contracts will be tested with a minimal test command and (optionally) by a list of test files to be executed (e.g., forge test ...testFiles --fail-fast).
• Make sure that your forge installation is up-to-date to enable --fail-fast.

Custom

If you set testingFramework to custom, SuMo will invoke the compile and test script defined in your package.json. This allows you to customize both scripts and have more control over the testing process. For example, you can define the scripts as follows:

//package.json
 scripts: {
    compile: "hardhat compile",
    test "hardhat test --bail && forge test --fail-fast"
 }

//sumo-config.js
 project: {
    buildDir: "artifacts",
 }

Additionally, you must also explicitly define a buildDir (matching your compile command) in your sumo-config.js.

⚠️ Limitations of Custom Test Scripts:
     * buildDir: must be explicitly specified it in the sumo-config.js
     * skipTests: will be ignored. You have to specify them in your custom script.

3) Mutation Testing Process Configuration

These fields allow you to further customize the mutation testing process:

CLI Usage

Selecting the Mutation Operators

Before starting the mutation process you can choose which mutation operators to use:

Viewing the available mutations

Running Mutation Testing

Viewing the results

SuMo automatically creates a sumo\results folder in the root directory of the project with the following reports:

• mutations.json: List of mutations in json format, synchronoysly updated during testing.
• index.html: A simple web display of the results (you can view this using VSCode extensions like Live Server). From here, you can also download a csv with the results.
• \mutants: Folder with mutated .sol source files (only if generated with sumo mutate)

Mutation Operators 👾

SuMo includes the following Traditional and Solidity-specific operators. Note that not all mutation operators are enabled by default.

Traditional Mutation Operators

Solidity Mutation Operators

Minimal Mutation Rules

Some mutation operators foresee a minimal version:

• The extended operators generate a more comprehensive set of mutants. These guarantee a more in-depth test adequacy assessment, but they can generate more than one replacement per target (e.g., + is mutated in both - and *), which can lead to longer execution times.
• The minimal operators define simplified rules that only inject one replacement per target (e.g., + is mutated in -), limiting the generation of subsumed mutants and speeding up the testing process.

By default, SuMo employs the extended operators. However, you can enable the minimal rules in the sumo-config.js file.

Publications

To cite SuMo, please use the following:

@article{BARBONI2022111445,
  title = {SuMo: A mutation testing approach and tool for the Ethereum blockchain},
  journal = {Journal of Systems and Software},
  volume = {193},
  pages = {111445},
  year = {2022},
  issn = {0164-1212},
  doi = {https://doi.org/10.1016/j.jss.2022.111445},
  author = {Morena Barboni and Andrea Morichetta and Andrea Polini}
}