Claude API returns 529 overloaded_error in production, what's the right way to handle it

Solved · 941 views · asked last month by jonah_b · edited

My app generates meal plans with the Anthropic API. A few times a day users get an error and I see this in the logs:

{"type":"error","error":{"type":"overloaded_error","message":"Overloaded"}}

Sometimes it even comes in the middle of a stream after text already started. Should I just retry? How many times? I don't want to hammer the API or double-charge myself.

What I’ve tried

Wrapped the call in a try/catch that retries once immediately. Sometimes works, sometimes the retry fails too.

Comment

Which SDK and version? The retry defaults matter here. chidi_eze last month · edited

3 answers

Marked as helpful by the asker

chidi_ezelast month · edited

Yes, retry, but with backoff, and let the SDK do it. The official SDK already retries 429, 5xx and 529 with exponential backoff (2 retries by default). Bump it:

const client = new Anthropic({ maxRetries: 4, timeout: 60_000 });

Mid-stream errors are different: the HTTP response was already a 200, so the SDK can't transparently retry. Handle it yourself: catch the error event, discard the partial text (or show "retrying..."), and start the request again.

Double charging: you pay for tokens that were actually generated. A failed request that produced nothing costs nothing; an interrupted stream costs what it produced. Small either way.

For a nicer UX when it keeps failing, fall back to a smaller model after the retries, a slightly different meal plan beats an error page.

Comment

didn't know the SDK already retries. my own retry was on top of it. removed mine, set maxRetries 4, added the mid-stream handling jonah_b last month · edited

The mid-stream part is so easy to miss. My UI showed half a message forever. sophie_l last month · edited

nils_twlast month · edited

Plus: log the request-id response header for failed calls. If it ever looks like more than the usual transient blips, that's what support will ask for.

Comment

sergio_ruizlast month · edited

Also cap your own concurrency. If a cron job fires 50 generations at the same moment, you're much more likely to hit 429s and 529s than with 5 at a time through a small queue. Smoothing your traffic helps more than extra retries.

Comment

Keyboard shortcuts

Claude API returns 529 overloaded_error in production, what's the right way to handle it

What I’ve tried

3 answers

Your answer

Claude API returns 529 overloaded_error in production, what's the right way to handle it

What I’ve tried

3 answers

Related

Your answer