Removing Near-Duplicates

Same fact, many copies

Knowledge bases are full of near-duplicates: FAQ copies, versioned pages, email quotes. Sending several copies wastes tokens, pushes out other useful content, and can make the model treat a point as more important than it is. Before packing, remove near-duplicates using a cheap similarity measure: shingle overlap (Jaccard) on word sequences works for copies; embedding similarity catches paraphrases. Keep the most authoritative or newest copy, not simply the first one returned.

Jaccard de-duplication on word shingles, run

I ran this with plain Python 3 (standard library only); the data is made-up example data. Two refund sentences share most 3-word shingles (similarity 0.62), so the second is dropped; the unrelated shipping sentence is kept. The 0.6 threshold is a tunable choice.

def shingles(t, k=3):
    w = t.lower().split()
    return {" ".join(w[i:i + k]) for i in range(len(w) - k + 1)}
def jaccard(a, b):
    return len(a & b) / len(a | b)
docs = {
    "d1": "Refunds are issued within 14 days of receiving the returned item.",
    "d2": "Refunds are issued within 14 days of receiving the returned item in good condition.",
    "d3": "Shipping to Pune takes three to five working days.",
}
kept = []
for name, text in docs.items():
    sims = [round(jaccard(shingles(text), shingles(docs[k])), 2) for k in kept]
    if any(s >= 0.6 for s in sims):
        print(name, "dropped as near-duplicate, similarity", max(sims))
    else:
        kept.append(name)
print("kept:", kept)

Output:

d2 dropped as near-duplicate, similarity 0.62
kept: ['d1', 'd3']

Choose which copy survives

When two chunks are duplicates, keep the one with the newer date or higher-authority source, and log what was dropped.

Quick check: What is a cost of sending duplicate chunks?

• They reduce latency
• They improve citations automatically
• They waste budget and can over-weight one source
• They shrink the model