My OpenAI streaming shows the whole answer at once at the end instead of word by word

Solved · 721 views · asked 3 months ago by yusuf_k · edited

Next.js App Router route handler. I set stream: true, the chat UI shows "..." for 10 seconds and then the full text appears. Locally too, not only on Vercel.

export async function POST(req: Request) {
  const stream = await openai.chat.completions.create({ model, messages, stream: true });
  let text = "";
  for await (const chunk of stream) text += chunk.choices[0]?.delta?.content ?? "";
  return Response.json({ text });
}

What I’ve tried

Windsurf changed the frontend to use a ReadableStream reader, still arrives at once.

Comment

You're collecting the chunks into text before returning, right? Look at the loop. mei_lin 3 months ago · edited

2 answers

Marked as helpful by the asker

mei_lin3 months ago · edited

Your server collects the whole stream into text and only then responds. Streaming from OpenAI to your server doesn't help if you don't pass it on. Return a stream:

export async function POST(req: Request) {
  const { messages } = await req.json();
  const completion = await openai.chat.completions.create({ model, messages, stream: true });
  const encoder = new TextEncoder();
  const body = new ReadableStream({
    async start(controller) {
      for await (const chunk of completion) {
        const t = chunk.choices[0]?.delta?.content;
        if (t) controller.enqueue(encoder.encode(t));
      }
      controller.close();
    },
  });
  return new Response(body, { headers: { "Content-Type": "text/plain; charset=utf-8" } });
}

On the client, read with res.body.getReader() and a TextDecoder, appending as chunks arrive. The Vercel AI SDK wraps all of this if you'd rather not do it by hand.

Comment

oh. I was streaming into a variable. works now, text comes in nicely yusuf_k 3 months ago · edited

dev_ana3 months ago · edited

If it still buffers after that: some proxies (and certain browser extensions) buffer text/plain. Rarely an issue on Vercel, but nginx needs X-Accel-Buffering: no.

Comment

good to know, I'll deploy behind nginx later yusuf_k 3 months ago · edited

Keyboard shortcuts

My OpenAI streaming shows the whole answer at once at the end instead of word by word

What I’ve tried

2 answers

Your answer

My OpenAI streaming shows the whole answer at once at the end instead of word by word

What I’ve tried

2 answers

Related

Your answer