Testing Your Tests? Who Watches the Watchmen?

Regardless of whether you’re working for a big corporation, a startup, or just for yourself, unit testing is not only helpful, but often indispensable. We use unit tests to test our code, but what happens if our tests are wrong or incomplete? What can we use to test our tests? Who watches the watchmen?

Enter Mutation Testing

No, no, it’s nothing like that. Mutation Testing ( or Mutant Analysis ) is a technique used to create and evaluate the quality of software tests. It consists of modifying the tests in very small ways. Each modified version is called a mutant and tests detect and reject mutants by causing the behavior of the original version to differ from the mutant. Mutations are bugs in our original code and analysis checks if our tests detect those bugs. In a nutshell, if a test still works after it’s mutated, it’s not a good test.

Mutation Testing with Humbug

Humbug is a mutation testing framework for PHP. In order for Humbug to be able to generate code coverage, we will have to have XDebug installed and enabled on our machine. Then, we can install it as a global tool.

composer global require 'humbug/humbug'

After this, if we run the

humbug

command, we should be able to see some of our Humbug installation information and an error indicating that we don’t have a humbug.json file.

Bootstrapping

Before we configure and use Humbug, we need a project that we can test. We will create a small PHP calculator package where we will run our unit and mutation tests. Let’s create a /Calculator folder. Inside it, let’s create our /src and /tests folders. Inside our /src folder, we will have our application code; the /tests folder will contain our unit tests. We will also need to use PHPUnit in our package. The best way to do that is using Composer. Let’s install PHPUnit using the following command:

composer global require phpunit/phpunit

Let’s create our Calculator. Inside the /src folder, create a Calculator.php file and add the following content:

<?php

namespace package\Calculator;

class Calculator {

    /**
     * BASIC OPERATIONS
     */
    public function add($a1, $a2) {
        return $a1 + $a2;
    }

    public function subtract($a1, $a2) {
        return $a1 - $a2;
    }

    public function multiply($a1, $a2) {
        return $a1 * $a2;
    }

    public function divide($a1, $a2) {

        if ($a2 === 0) {
            return false;
        }

        return $a1 / $a2;
    }

    /*
     * PERCENTAGE
     */

    //This will return $a1 percent of $a2
    public function percentage($a1, $a2) {
        return ( $a1 / $a2 ) * 100;
    }

    /*
     * PI
     */

    //Returns the value of pi
    public function pi() {
        return pi();
    }

    /*
     * LOGARITHMIC
     */

    //Returns the basic logarithm in base 10
    public function log($a) {
        return log10($a);
    }

}

It is a rather straightforward program. A simple calculator, with the basic arithmetic, percentage and logarithmic operations and a function to return the value of pi. Next, inside our /tests folder, let’s create the unit tests for our calculator. If you need help with unit testing in PHP, check out this tutorial. Create a CalculatorTest.php file and add the following:

<?php

use package\Calculator\Calculator;

class CalculatorTest extends PHPUnit_Framework_TestCase {

    public function testAdd() {
        $calculator = new Calculator();
        $result = $calculator->add(2, 3);
        $this->assertEquals($result, 5);
    }

    public function testSubtract() {
        $calculator = new Calculator();
        $result = $calculator->subtract(6, 3);
        $this->assertEquals($result, 3);
    }

    public function testMultiply() {
        $calculator = new Calculator();
        $result = $calculator->multiply(6, 3);
        $this->assertEquals($result, 18);
    }

    public function testDivide() {
        $calculator = new Calculator();
        $result = $calculator->divide(6, 3);
        $this->assertEquals($result, 2);
    }

}

This will be our initial test stack. If we run the phpunit, command we will see that it executes successfully, and our 4 tests and 4 assertions will pass. It is important that all of our tests are passing, otherwise, Humbug will fail.

Configuring Humbug

Humbug may either be configured manually, by creating a humbug.json.dist file, or automatically, by running the command:

humbug configure

Running the command will ask us for answers to some questions:

• What source directories do you want to include? In this one we will go with src/, the directory of our source code.
• Any directories you want to exclude from within your source directory? May be useful in some cases, like an external vendor directory that we don’t want tested. It does not apply in our current case.
• Single test suite timeout in seconds. Let’s go with 30 seconds on this one. It is probably too much, but we want to be sure everything has had enough time to run.
• Where do you want to store your text log? humblog.txt comes as default and we will leave it as that.
• Where do you want to store your json log (if you need it)? The default comes empty but we will store it in humblogjson.json.
• Generate “humblog.json.dist”? This file will, when generated, contain all the configuration values we just supplied. We can edit it manually if we want to change something.

Using Humbug

Now that we have both our application running with tests and Humbug installed, let’s run Humbug and check the results.

   humbug

The result should be close to this:

Interpreting Humbug results

The number of mutations created is just the number of small changes introduced by Humbug to test our tests. A killed mutant (.) is a mutation that caused a test to fail. Don’t be confused, this is a positive result! An escaped mutation (M) is a mutation where the test still passed. This is not a positive result, we should go back to our test and check what’s missing. An uncovered mutation (S) is a mutation that occurs in a line not covered by a unit test. Fatal errors (E) and timeouts (T) are mutations that created fatal errors and mutations that create infinite loops, respectively.

What about the metrics?

The Mutation Score Indicator indicates the percentage of generated mutations that were detected. We want to aim at 100%. Mutation Code Coverage indicates the percentage of tests covered by mutations. The Mutation Score Indicator gives you some idea of how effective the tests that do exist really are. Analyzing our humbug log, we can see that we have 9 mutants not covered, and some really bad metrics. Take a look at the humblogjson.json file. This file was generated automatically just like the humblog.txt file, and contains much more detailed information on what failed, where and why. We haven’t tested our percentage, pi and logarithm functions. Also, we need to cover the case where we divide a number by 0. Let’s add some more tests to cover the missing situations:

    public function testDivideByZero() {
        $calculator = new Calculator();
        $result = $calculator->divide(6, 0);
        $this->assertFalse($result);
    }

    public function testPercentage() {
        $calculator = new Calculator();
        $result = $calculator->percentage(2, 50);
        $this->assertEquals($result, 4);
    }

    public function testPi() {
        $calculator = new Calculator();
        $result = $calculator->pi();
        $this->assertEquals($result, pi());
    }

    public function testLog() {
        $calculator = new Calculator();
        $result = $calculator->log(10);
        $this->assertEquals($result, 1);
    }

This time around, 100% means that all mutations were killed and that we have full code coverage.

Downsides

The biggest downside of mutation testing, and by extension Humbug, is performance. Mutation testing is a slow process as it depends on a lot of factors like interplay between lines of code, number of tests, level of code coverage, and the performance of both code and tests. Humbug also does initial test runs, logging and code coverage, which add to the total duration. Additionally, Humbug is PHPUnit specific, which can be a problem for those who are using other testing frameworks. That said, Humbug is under active development and will continue to improve.

Conclusion

Humbug can be an important tool for maintaining your app’s longevity. As the complexity of your app increases, so does the complexity of your tests – and having them all at 100% all the time becomes incredibly important, particularly when dealing with enterprise ecosystems. The code we used in this tutorial can be cloned here. Have you used Humbug? Do you do mutation testing another way? Give us your thoughts on all this!

Frequently Asked Questions (FAQs) about “Who Watches the Watchmen?”