Managing Conversation History

Sliding window plus summary

Sending the entire conversation every turn eventually overflows the window and grows cost every turn. Common strategies: a sliding window (keep the last N turns verbatim), a running summary of older turns (goals, decisions, facts, open questions), or both. Summaries should be updated incrementally and checked, because a lost detail (an account number, a constraint the user stated) causes confusing errors later. Keep important facts in a structured state (fields, not prose) when your application can.

Long conversations and agents

Multi-turn chats and agents keep adding text; you must decide what to keep, summarise or fetch on demand.

Keeping the last four turns plus a summary, run

I ran this with plain Python 3 (standard library only); the data is made-up example data. Twelve turns become five context items: one summary placeholder for turns 1 to 8 and the last four turns verbatim. In a real system the summary would be written by a model and stored.

turns = [f"turn {i}: user/assistant exchange" for i in range(1, 13)]
keep_last = 4
older, recent = turns[:-keep_last], turns[-keep_last:]
summary = f"[summary of turns 1-{len(older)}: goals, decisions, open questions]"
context = [summary] + recent
print("original turns:", len(turns))
print("sent to model:", len(context), "items")
for line in context:
    print(" ", line)

Output:

original turns: 12
sent to model: 5 items
  [summary of turns 1-8: goals, decisions, open questions]
  turn 9: user/assistant exchange
  turn 10: user/assistant exchange
  turn 11: user/assistant exchange
  turn 12: user/assistant exchange

Pin user-stated constraints

Store explicit user constraints (budget, language, deadlines) in a pinned state block that is never summarised away.

Quick check: Why not send the full conversation forever?

• It makes answers shorter
• Models cannot read history
• History is illegal to store
• It eventually overflows the window and increases cost every turn