AskVibeMarked as helpful by the asker
You're calling a blocking client inside an async def. That freezes the event loop, the single thread that serves all requests, for 30 seconds.
Two correct fixes, pick one:
A. Use the async client (preferred):
from openai import AsyncOpenAI
client = AsyncOpenAI() # create once, at module level
@app.post("/summarize")
async def summarize(doc: Doc):
resp = await client.chat.completions.create(model="gpt-4o-mini", messages=[...])
return {"summary": resp.choices[0].message.content}B. Make the endpoint plain def. FastAPI then runs it in a threadpool, so it doesn't block the loop.
What you must not do is mix: async def + sync library. Same applies to requests, time.sleep, sync database drivers and so on. Also create the client once instead of per request.
Switched to AsyncOpenAI, 10 parallel requests now and /health answers instantly. I had no idea async def could make it worse. chen_wei · edited
The 'must not mix' rule should be printed on the FastAPI homepage rosa_m · edited
It kind of is, in the 'Concurrency and async / await' docs page. Nobody reads it until this happens :) grace_mw · edited