# Detecting Test Tampering — Safe Autonomous Code Fixing

Source: https://www.skillbyai.com/en/safe-autonomous-code-fixing/g-tamper

> Removed assertions, skips and trivial tests.

## Look at what changed in tests

The most common way to "fix" a failing test without fixing the code is to change the test: delete assertions, add skip or xfail markers, loosen expected values, or replace checks with assertions that are always true. A tampering detector scans the patch for these patterns. Simplest policy for autonomous fixes: **test files are read-only**; tests are added or changed only in a separate, human-reviewed step.

## Block the cheats and the leaks

Mechanical checks stop test tampering, edits to protected files and leaked secrets.

![Three ideas: test tampering, protected paths, secrets.](assets/figures/safe-autonomous-code-fixing/section-5-map.svg) — Figure 5.1 — Tampering, protected paths and secrets.

## Scanning patches for test tampering, run

I ran this with Python 3 (standard library) and, where it uses git, real git in a throwaway temporary repository. Candidate patches are written by hand to stand in for model output. An honest one-line code fix is clean. A cheating patch is flagged three times: an assertion removed, a skip decorator added, and a trivial assertTrue(True) added. Pattern checks catch common cases; making tests read-only closes the loophole entirely.

```python
import re
def review_test_changes(diff):
    findings = []
    for line in diff.splitlines():
        if line.startswith("-") and re.search(r"\bassert", line) and not line.startswith("---"):
            findings.append("assertion removed: " + line[1:].strip())
        if line.startswith("+") and re.search(r"@(unittest\.)?skip|pytest\.mark\.(skip|xfail)", line):
            findings.append("test skipped: " + line[1:].strip())
        if line.startswith("+") and re.search(r"assert\w*\(.*\b(True|1)\)\s*$", line):
            findings.append("trivial assertion added: " + line[1:].strip())
    return findings
honest = """--- a/pricing.py
+++ b/pricing.py
-    if code == "FLAT50" and total > 50:
+    if code == "FLAT50" and total >= 50:"""
cheating = """--- a/test_pricing.py
+++ b/test_pricing.py
-        self.assertEqual(discount(50, "FLAT50"), 0)
+    @unittest.skip("flaky")
+        self.assertTrue(True)"""
for name, diff in [("honest patch", honest), ("cheating patch", cheating)]:
    f = review_test_changes(diff)
    print(name, "->", "clean" if not f else f)
```

Output:

```
honest patch -> clean
cheating patch -> ['assertion removed: self.assertEqual(discount(50, "FLAT50"), 0)', 'test skipped: @unittest.skip("flaky")', 'trivial assertion added: self.assertTrue(True)']
```

## Make tests read-only for bots

Enforce at the file level (protected paths) instead of relying only on pattern detection.

**Quiz:** Which change should a tampering check flag?

- [ ] Renaming a local variable in source
- [ ] Changing > to >= in source code
- [ ] Adding a docstring to a function
- [x] Adding @unittest.skip to the failing test

*Answer:* Adding @unittest.skip to the failing test. Skipping a test hides the failure.