पाठ 9 / 35

Log Hygiene: Volume, Privacy and Sampling

यह पाठ अभी हिंदी में अनुवादित नहीं है, इसलिए अंग्रेज़ी में दिखाया जा रहा है।

Keep logs useful and affordable by controlling volume, redacting personal data and sampling repetitive events.

लेखक: · 15 साल से वेब डेवलपर

More logs is not better logs

Log cost grows linearly with volume: every byte is shipped, indexed, stored and backed up. A single noisy INFO line inside a loop that runs a thousand times per request can dominate a company's logging bill.

Three habits keep logs healthy. First, log decisions and outcomes, not every step: one line per request with its result and duration (an access log or wide event) is often more useful than ten partial lines. Second, sample repetitive events: log every error, but maybe 1 in 100 successful health checks, and record the sample rate as a field so counts can be scaled back up. Third, never log secrets or unnecessary personal data: passwords, tokens, full card numbers, Aadhaar or PAN numbers, OTPs. Once such data reaches a log store, it is copied to backups and readable by everyone with log access; under data-protection laws such as India's DPDP Act, that can be a reportable problem.

Redaction should happen as early as possible, ideally in the application, with a second filter in the collector as a safety net.

Redacting before the line is written

A handler-level filter masks card numbers and email addresses in the final message. Attaching it to the handler means it runs for every record that handler emits.

import logging, re, sys

CARD = re.compile(r"\b(\d{4})\d{8}(\d{4})\b")
EMAIL = re.compile(r"([A-Za-z0-9._%+-])[A-Za-z0-9._%+-]*@")

class RedactFilter(logging.Filter):
    def filter(self, record):
        msg = record.getMessage()
        msg = CARD.sub(r"\1********\2", msg)
        msg = EMAIL.sub(r"\1***@", msg)
        record.msg, record.args = msg, None
        return True

h = logging.StreamHandler(sys.stdout)
h.addFilter(RedactFilter())
log = logging.getLogger("pay")
log.addHandler(h)
log.setLevel(logging.INFO)

log.info("payment by %s card=%s", "riya.sharma@example.com", "4111111111111111")

Output:

payment by r***@example.com card=4111********1111

The photocopy problem

If you photocopy your Aadhaar card for one form, that copy can end up in many files you never see again. A secret written to a log is the same: it is copied to the agent buffer, the log store, replicas and backups. Removing it later means finding every copy.

त्वरित जाँच: A service logs every successful health check, producing most of its log volume. What is the best fix?

  • Delete the health-check endpoint
  • Raise the whole service's log level to ERROR
  • Drop or sample successful health-check lines and keep logging failures
  • Move the logs to a bigger disk
Answer

Drop or sample successful health-check lines and keep logging failures — Sampling or dropping the repetitive success lines cuts volume while failures stay fully visible. Raising the level to ERROR would also hide useful INFO events.