Coverage and Mutation Testing

Measuring how well tests test

Code coverage reports which lines and branches your tests executed. JaCoCo is the standard Java tool, producing HTML and XML reports through Maven or Gradle plugins and integrating with CI and code-review tools. Coverage is useful for finding untested code, especially untested branches in important logic. But coverage measures execution, not checking: a test that calls a method and asserts nothing still counts. That is why coverage targets such as "90% or the build fails" often lead to low-value tests. Mutation testing measures test strength: a tool such as PIT (pitest) makes small changes (mutants) to your code, for example flipping >= to >, replacing a return value or removing a call, and runs the tests; if no test fails, the mutant survived, revealing a gap in your assertions. Mutation testing is slower, so run it on core modules or nightly. Use both as diagnostics: look at uncovered branches and surviving mutants in critical code and decide whether they matter.

Executed versus checked

Coverage shows which code ran; mutation testing shows whether tests notice when that code changes.

JaCoCo and PIT in Maven

Coverage report on every build; mutation testing on demand.

<!-- versions are examples; use the latest stable releases -->
<build>
  <plugins>
    <plugin>
      <groupId>org.jacoco</groupId>
      <artifactId>jacoco-maven-plugin</artifactId>
      <version>0.8.13</version>
      <executions>
        <execution><goals><goal>prepare-agent</goal></goals></execution>
        <execution>
          <id>report</id>
          <phase>verify</phase>
          <goals><goal>report</goal></goals>
        </execution>
      </executions>
    </plugin>
    <plugin>
      <groupId>org.pitest</groupId>
      <artifactId>pitest-maven</artifactId>
      <version>1.19.1</version>
      <dependencies>
        <dependency>
          <groupId>org.pitest</groupId>
          <artifactId>pitest-junit5-plugin</artifactId>
          <version>1.2.2</version>
        </dependency>
      </dependencies>
      <configuration>
        <targetClasses><param>com.shop.pricing.*</param></targetClasses>
      </configuration>
    </plugin>
  </plugins>
</build>

<!-- mvn verify                                -> target/site/jacoco/index.html
     mvn org.pitest:pitest-maven:mutationCoverage -> target/pit-reports/ -->

Surviving mutants point to missing assertions

If PIT changes total >= 100_000 to total > 100_000 and every test still passes, you have no test at exactly the boundary. That is a concrete, valuable test to add.

Quick check: What does a surviving mutant in mutation testing indicate?

• The code has a bug
• Coverage is 100%
• The tests did not detect that change, revealing a gap in what they check
• The build is broken