Mutation testing

BulkPrice has 100% line coverage. Every line runs in a test, and the badge in the README is green. Then Infection plants five small bugs, one at a time, and the tests pass with every one of them.

Coverage measures where tests go. Mutation testing measures what they check. You've been doing it by hand since 4.1: every planted bug in this world is a mutant. Infection is the tool that plants them for you.

What Infection does

Infection reads your code and applies mutators: small, mechanical changes a typo or a bad merge could make. Each change is one mutant. Infection runs your tests against every mutant, using the coverage report to pick only the tests that reach the changed line.

Mutator	Change
GreaterThanOrEqualTo	>= becomes >
IncrementInteger, DecrementInteger	90 becomes 91 or 89
LogicalAnd	&& becomes ||
ReturnRemoval	A return disappears
FunctionCallRemoval	A call like sort($low) disappears

A mutant is killed when a test fails, escaped when every test still passes, and not covered when no test even runs its line.

Escaped mutants:
================

1) src/BulkPrice.php:17    [M] GreaterThanOrEqualTo

-        if ($quantity >= 50) {
+        if ($quantity > 50) {

Read each escaped mutant as a question: "Which test would fail if someone made this change?" Here the answer is none, because no test has exactly 50 units in it. The fix is a test, not a change to the code.

Reading the score

Infection reports a Mutation Score Indicator (MSI): the share of mutants that were killed. It also reports a covered MSI, counting only the mutants on lines your tests run. Covered MSI is the honest measure of how good your assertions are.

In CI, run it with thresholds, and only on what changed, so it stays fast:

vendor/bin/infection --min-msi=80 --min-covered-msi=90 --git-diff-lines --threads=max

Infection needs a coverage driver, PCOV or Xdebug. The exercises in this world plant their mutants by hand, so they run without one.

The coverage badge	100%
Infection	5 mutants escaped

Equivalent mutants

Some mutants can't be killed, because they don't change behaviour:

return $quantity > 0 ? $unitPence * $quantity : 0;

Changing > to >= makes no difference: at 0 units both branches give 0. That's an equivalent mutant. Don't write a strange test to kill it. Ignore that mutator for that method in Infection's config, with a comment saying why. 100% MSI is rarely the goal. A high covered MSI, with every survivor understood, is.

<?php

use PHPUnit\Framework\TestCase;

// Line coverage: 100%. Mutants killed: 0 of 5.
final class BulkPriceTest extends TestCase
{
    public function testOrdersOfTenOrMoreAreDiscounted(): void
    {
        self::assertLessThan(2000, BulkPrice::total(100, 20));
    }

    public function testOrdersOfFiftyOrMoreAreDiscountedMore(): void
    {
        self::assertLessThan(5400, BulkPrice::total(100, 60));
    }
}

<?php

use PHPUnit\Framework\TestCase;

// Line coverage: 100%. Mutants killed: 5 of 5.
final class BulkPriceTest extends TestCase
{
    public function testExactlyTenUnitsGetTenPercentOff(): void
    {
        self::assertSame(900, BulkPrice::total(100, 10));
    }

    public function testExactlyFiftyUnitsGetTwentyPercentOff(): void
    {
        self::assertSame(4000, BulkPrice::total(100, 50));
    }

    public function testTheDiscountRoundsDownToThePenny(): void
    {
        self::assertSame(980, BulkPrice::total(99, 11));
    }
}