Selecting Few-Shot Examples per Request

Dynamic few-shot

Instead of a fixed set of examples on every call, keep a pool of labelled examples and select the few most similar to the current request (by embeddings or keyword overlap). This makes examples more relevant, keeps the prompt shorter, and lets the pool grow without growing the prompt. Make sure the selection still covers edge cases and does not always pick the same easy examples, and keep the pool reviewed: a wrong example in the pool now gets shown exactly when it is most persuasive.

Choosing examples by similarity, run

I ran this with plain Python 3 (standard library only); the data is made-up example data. For the query about not remembering a password, bag-of-words cosine similarity picks the two password examples (0.60 and 0.55) from a pool of five.

import math, collections
def vec(t):
    return collections.Counter(t.lower().replace("?", "").split())
def cos(a, b):
    num = sum(a[w] * b[w] for w in a)
    return num / (math.sqrt(sum(v * v for v in a.values())) * math.sqrt(sum(v * v for v in b.values())))
pool = [
    "how do I reset my password",
    "my card was charged twice",
    "the app crashes on startup",
    "can I change my billing address",
    "I forgot my login password",
]
query = "I cannot remember my password"
scores = sorted(((round(cos(vec(query), vec(p)), 2), p) for p in pool), reverse=True)
for s, p in scores[:2]:
    print(s, "|", p)

Output:

0.6 | I forgot my login password
0.55 | how do I reset my password

Mix similar and diverse

Take the top two similar examples plus one fixed hard case so the model still sees the format boundary.

Quick check: What is dynamic few-shot selection?

• Choosing the examples most similar to each request from a larger pool
• Using no examples ever
• Fine-tuning on every request
• Sending the whole pool every time