Allocating a Token Budget

Reserve, allocate, trim in order

Start from the window size, reserve output tokens for the longest answer you allow, and give every block a budget. When the request is too large, trim by an explicit priority: never drop the system rules or the current question; trim documents to the best few, then summarise or cut older history, then reduce examples. Writing this as code makes behaviour predictable and testable, instead of an API error or silent truncation in production. Log when trimming happens: frequent trimming is a signal to improve retrieval or summarisation.

Decide the split before you run out

A token budget turns overflow from a crash into a policy.

A budget with trimming priority, run

I ran this with plain Python 3 (standard library only); the data is made-up example data. An 8,000-token window with 1,000 reserved for output leaves 7,000 for input. The request asks for 8,620, so the code trims documents by 1,620 and leaves system rules, examples, history and question untouched.

window = 8000          # model context window (tokens)
reserve_output = 1000  # room for the answer
parts = {"system": 600, "examples": 1200, "history": 2600, "documents": 4100, "question": 120}
budget = window - reserve_output
used = sum(parts.values())
print("budget for input:", budget, "| requested:", used, "| over by:", used - budget)
# trim in priority order: documents first, then history; never system or question
for name in ["documents", "history", "examples"]:
    over = sum(parts.values()) - budget
    if over <= 0:
        break
    cut = min(over, parts[name])
    parts[name] -= cut
    print(f"trim {name:<9} by {cut}")
print("final:", parts, "| total:", sum(parts.values()))

Output:

budget for input: 7000 | requested: 8620 | over by: 1620
trim documents by 1620
final: {'system': 600, 'examples': 1200, 'history': 2600, 'documents': 2480, 'question': 120} | total: 7000

Test the overflow path

Write a test with an oversized history and documents; the overflow path is the one that breaks in production.

Quick check: What should normally never be trimmed?

• Low-scoring documents
• The system rules and the current question
• Very old history turns
• Extra examples