prompt is too long: 213440 tokens > 200000 maximum, my chat app breaks for long conversations

Solved · 762 views · asked 4 weeks ago by sanne_dev · edited

I send the whole conversation history to the Claude API on every message. Power users with long chats now get:

400 invalid_request_error: prompt is too long: 213440 tokens > 200000 maximum

What's the normal way to deal with this? Cut old messages? It also costs a lot per message for those users.

What I’ve tried

Cutting to the last 50 messages. Works mostly, but some messages contain pasted documents and 50 is still too much. And the bot forgets things users told it early on.

Comment

Are you sending tool results too, or just messages? sergio_ruiz 4 weeks ago · edited

Just messages, but some have pasted docs of 20k tokens. sanne_dev 4 weeks ago · edited

3 answers

Marked as helpful by the asker

sergio_ruiz4 weeks ago · edited

Budget by tokens, not by message count, and summarize what you drop:

Keep the system prompt + a running summary + the most recent messages that fit a budget (say 60k tokens). Count with the token counting endpoint or estimate ~4 chars per token and leave margin.
When messages fall out of the window, ask a cheap model to fold them into the running summary ("facts the user told us, decisions made, open questions"). Store that summary on the conversation row.
Pasted documents: don't resend them forever. Summarize them once or store them and only include them when relevant.
Use prompt caching on the stable prefix (system prompt + summary). It cuts cost a lot for long chats.

This fixes the error, the cost, and the "forgets early stuff" problem at the same time.

Comment

The running summary stored on the conversation is smart. Implemented with a 60k budget, costs for the heavy users dropped by more than half. sanne_dev 4 weeks ago · edited

zainab_a4 weeks ago · edited

Don't forget tool results if you use tools. In my app they were 70% of the context. Truncating large tool outputs to the relevant part was the biggest win.

Comment

leo_prompts4 weeks ago · edited

We combine a sliding window with retrieval: old messages get embedded, and for each new message we pull the 3–5 most relevant old ones back in. Works well when users refer back to specific things ("the recipe from last week"). More moving parts than a summary though, so start with the summary.

Comment

Keyboard shortcuts

prompt is too long: 213440 tokens > 200000 maximum, my chat app breaks for long conversations

What I’ve tried

3 answers

Your answer