Secret Scanning

Patterns, entropy and verification

Secret scanners such as Gitleaks, TruffleHog and the scanning built into code hosts look for credentials in code, history and sometimes logs. They use three techniques. Patterns: many providers use recognisable prefixes, so a regular expression like AKIA[0-9A-Z]{16} finds AWS access key IDs. Entropy: random-looking strings have high Shannon entropy, which catches generic tokens without a known prefix, at the cost of more false positives. Verification: some tools try the found credential against the provider's API to tell you whether it is live, which helps you prioritise. Run scanning in three places: a pre-commit hook to stop the mistake locally, a CI job on every pull request because hooks can be skipped, and push protection on the code host, which blocks a push containing a known secret format. Also scan full history once when adopting the tool, because old leaks still matter.

A toy scanner: pattern plus entropy

Real scanners are far more complete, but the mechanics look like this. The key below is the documented AWS example value, not a real credential.

import math, re
from collections import Counter

def entropy(s):
    n = len(s)
    return -sum(c / n * math.log2(c / n) for c in Counter(s).values())

AWS_ID = re.compile(r"AKIA[0-9A-Z]{16}")
lines = ['region = "ap-south-1"',
         'aws_key = "AKIAIOSFODNN7EXAMPLE"',
         'token = "q8Zr2LmX9vTn4KpW7sYb3HdF"']
for line in lines:
    value = line.split('"')[1]
    hit = "pattern" if AWS_ID.search(line) else ("entropy" if entropy(value) > 4.0 else "-")
    print(f"{entropy(value):.2f}  {hit:<8} {line}")

Output:

3.12  -        region = "ap-south-1"
3.68  pattern  aws_key = "AKIAIOSFODNN7EXAMPLE"
4.58  entropy  token = "q8Zr2LmX9vTn4KpW7sYb3HdF"

Allow-list false positives explicitly

Test fixtures and example keys will trigger findings. Use the scanner's allow-list or inline ignore comment with a reason, rather than turning the rule off for the whole repository.

त्वरित जाँच: Why run a secret scan in CI if developers already have a pre-commit hook?

• CI scanners are always more accurate
• Pre-commit hooks cannot read files
• Hooks are local and can be skipped or not installed, so CI is the enforced check
• CI scans are required by Git