Stream LLM Responses to the Browser Without Losing State

For: A mid-level full-stack engineer at a seed-stage SaaS startup who has wired up a working OpenAI chat feature but is now getting user complaints that the UI freezes for 8–12 seconds before the full response appears — and every tutorial they find either demos a toy Next.js app with no auth, no per-user session state, and no explanation of what happens when the stream breaks mid-response

To stream LLM responses to the browser without losing conversation state or token accounting, do the persistence on the server inside the same async generator that yields tokens to the client — not on the client after the stream closes. That single decision fixes three problems at once: a UI that freezes for 8–12 seconds, usage data lost on client disconnects, and race conditions between your chat store and your billing table.

This tutorial walks through a real implementation with FastAPI, OpenAI's streaming API, Server-Sent Events, and a React client. Roughly 250 lines of code end-to-end.

Why your current setup freezes

If you are calling openai.chat.completions.create() without stream=True, the SDK blocks until the full completion arrives. For a 400-token answer on GPT-4-class models, that is commonly 6–12 seconds of dead UI. Switching to stream=True and forwarding chunks fixes perceived latency, but naive implementations introduce three new problems:

• The assistant message never gets written to your database if the user closes the tab mid-stream.
• Token usage for billing is only reported in the final chunk — if you drop the connection, you drop the invoice.
• If you POST the assembled text back from the client after streaming, you now have two writers (server stream + client POST) racing on the same message row.

The fix: one server-side async generator that both yields to the wire and writes to Postgres in the same pass.

Prerequisites

• Python 3.11+, Node 20+
• An OpenAI API key with access to a chat model
• Postgres (or any store — SQLite works for the tutorial)
• Familiarity with async Python and React hooks

Install:

pip install fastapi uvicorn openai sqlalchemy asyncpg python-dotenv
npm create vite@latest chat-client -- --template react-ts

Step 1: Model the conversation state

Two tables. Messages, and a running token counter per user. Keep it boring.

from sqlalchemy import Column, String, Integer, DateTime, ForeignKey, Text
from sqlalchemy.orm import declarative_base
import uuid, datetime

Base = declarative_base()

class Message(Base):
    __tablename__ = "messages"
    id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
    user_id = Column(String, index=True, nullable=False)
    conversation_id = Column(String, index=True, nullable=False)
    role = Column(String, nullable=False)  # user | assistant
    content = Column(Text, nullable=False, default="")
    prompt_tokens = Column(Integer, default=0)
    completion_tokens = Column(Integer, default=0)
    status = Column(String, default="complete")  # streaming | complete | interrupted
    created_at = Column(DateTime, default=datetime.datetime.utcnow)

Note the status column. When the client disconnects mid-stream, we mark the row interrupted rather than deleting or leaving it half-written. Frontend can render an inline "response was cut off" state.

Step 2: Insert the assistant row before streaming starts

Reserve the row up front. This gives you a stable message ID to reference from the SSE stream and prevents the "where do I write partial output" question.

async def create_pending_assistant_message(session, user_id, conversation_id):
    msg = Message(
        user_id=user_id,
        conversation_id=conversation_id,
        role="assistant",
        content="",
        status="streaming",
    )
    session.add(msg)
    await session.commit()
    return msg.id

Step 3: The streaming endpoint

Here is the core pattern. One async generator does three things: yields SSE frames to the client, accumulates tokens in memory, and writes to the database when the stream ends or when the client disconnects.

from fastapi import FastAPI, Depends, Request
from fastapi.responses import StreamingResponse
from openai import AsyncOpenAI
import json, asyncio

app = FastAPI()
client = AsyncOpenAI()

async def stream_completion(request: Request, user_id: str, conversation_id: str, prompt: str, session):
    # 1. Reserve the assistant row
    message_id = await create_pending_assistant_message(session, user_id, conversation_id)

    # 2. Load prior turns for context
    history = await load_history(session, conversation_id)
    history.append({"role": "user", "content": prompt})

    buffer = []
    prompt_tokens = 0
    completion_tokens = 0
    final_status = "complete"

    try:
        stream = await client.chat.completions.create(
            model="gpt-4o-mini",
            messages=history,
            stream=True,
            stream_options={"include_usage": True},
        )

        # Send the message_id first so client can render a placeholder
        yield f"event: start\ndata: {json.dumps({'message_id': message_id})}\n\n"

        async for chunk in stream:
            if await request.is_disconnected():
                final_status = "interrupted"
                break

            if chunk.choices and chunk.choices[0].delta.content:
                token = chunk.choices[0].delta.content
                buffer.append(token)
                yield f"event: token\ndata: {json.dumps({'t': token})}\n\n"

            if chunk.usage:
                prompt_tokens = chunk.usage.prompt_tokens
                completion_tokens = chunk.usage.completion_tokens

        yield f"event: done\ndata: {json.dumps({'status': final_status})}\n\n"

    except Exception as e:
        final_status = "error"
        yield f"event: error\ndata: {json.dumps({'message': str(e)})}\n\n"
    finally:
        # 3. Always persist what we got, even on disconnect
        await session.execute(
            update(Message)
            .where(Message.id == message_id)
            .values(
                content="".join(buffer),
                prompt_tokens=prompt_tokens,
                completion_tokens=completion_tokens,
                status=final_status,
            )
        )
        await session.commit()

@app.post("/chat/stream")
async def chat_stream(request: Request, body: ChatRequest, user=Depends(current_user), session=Depends(db)):
    return StreamingResponse(
        stream_completion(request, user.id, body.conversation_id, body.prompt, session),
        media_type="text/event-stream",
        headers={"X-Accel-Buffering": "no", "Cache-Control": "no-cache"},
    )

Three details that matter:

• stream_options={"include_usage": True} — without this, OpenAI does not send a usage block in the streaming response and you cannot bill accurately. It arrives in the final chunk.
• request.is_disconnected() — the reason the finally block still writes partial content. If your user closes the tab at token 200 of a 400-token response, you keep the 200 tokens and mark it interrupted.
• X-Accel-Buffering: no — if you deploy behind Nginx or any reverse proxy, it will buffer SSE by default and re-freeze the UI. This header disables that.

Step 4: The React client

Use the Fetch API with a ReadableStream reader. EventSource is simpler but cannot send POST bodies or auth headers cleanly.

import { useState, useRef } from "react";

export function useChatStream() {
  const [tokens, setTokens] = useState("");
  const [status, setStatus] = useState<"idle" | "streaming" | "done" | "interrupted">("idle");
  const abortRef = useRef(null);

  async function send(prompt: string, conversationId: string) {
    setTokens("");
    setStatus("streaming");
    const controller = new AbortController();
    abortRef.current = controller;

    const res = await fetch("/chat/stream", {
      method: "POST",
      headers: { "Content-Type": "application/json", Authorization: `Bearer ${getToken()}` },
      body: JSON.stringify({ prompt, conversation_id: conversationId }),
      signal: controller.signal,
    });

    const reader = res.body!.getReader();
    const decoder = new TextDecoder();
    let buf = "";

    while (true) {
      const { value, done } = await reader.read();
      if (done) break;
      buf += decoder.decode(value, { stream: true });

      // Parse SSE frames: split on double newline
      const frames = buf.split("\n\n");
      buf = frames.pop() ?? "";

      for (const frame of frames) {
        const eventMatch = frame.match(/^event: (.+)$/m);
        const dataMatch = frame.match(/^data: (.+)$/m);
        if (!eventMatch || !dataMatch) continue;
        const payload = JSON.parse(dataMatch[1]);

        if (eventMatch[1] === "token") setTokens(prev => prev + payload.t);
        if (eventMatch[1] === "done") setStatus(payload.status);
      }
    }
  }

  function cancel() {
    abortRef.current?.abort();
  }

  return { tokens, status, send, cancel };
}

Manual SSE parsing looks tedious, but it is 20 lines and gives you POST + auth + cancellation. That trade beats the fifteen-minute debugging session when EventSource silently reconnects and re-triggers your billing.

Step 5: Test disconnect recovery

Start the server, open the client, send a long prompt ("write a 500-word essay about…"), and close the tab mid-stream. Then query:

SELECT id, status, length(content), completion_tokens
FROM messages
WHERE role = 'assistant'
ORDER BY created_at DESC LIMIT 5;

Expected output:

 id        | status      | length | completion_tokens
-----------+-------------+--------+-------------------
 8a3f...   | interrupted | 847    | 0
 91b2...   | complete    | 2103   | 421

The interrupted row has partial content but zero completion tokens — because OpenAI only sends usage in the final chunk, which never arrived. For billing, this is actually the correct behavior: you were not charged for the tokens after the disconnect either. If you want to bill for what did stream, estimate with tiktoken on the buffered content before writing:

import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o-mini")
if final_status == "interrupted":
    completion_tokens = len(enc.encode("".join(buffer)))

Step 6: Resume from an interrupted message

When the user returns, the frontend should be able to show the partial response and offer a "continue" button. Server side:

@app.post("/chat/resume/{message_id}")
async def resume(message_id: str, request: Request, session=Depends(db)):
    msg = await session.get(Message, message_id)
    if msg.status != "interrupted":
        raise HTTPException(400, "Message is not interrupted")
    # Re-issue completion with prior partial as an assistant turn ending mid-sentence,
    # asking the model to continue from where it left off.
    ...

Whether to resume or regenerate is a product decision. Regenerate is safer (no weird sentence boundaries). Resume is cheaper. Most teams pick regenerate and let power users retry.

Step 7: Wire billing to the same write

Because token counts are written in the same transaction as the message, your usage aggregation query is a single join:

SELECT user_id,
       SUM(prompt_tokens) AS prompt,
       SUM(completion_tokens) AS completion
FROM messages
WHERE created_at >= date_trunc('month', now())
GROUP BY user_id;

No separate usage table. No reconciliation job. If the message row exists, the tokens are accounted for.

Common errors

UI still freezes for several seconds before tokens appear

Almost always proxy buffering. Check for Nginx, Cloudflare, or a load balancer between the client and FastAPI. Set proxy_buffering off; in Nginx and add X-Accel-Buffering: no to the response. On Cloudflare, ensure the route is not going through their default HTML minification/buffering.

request.is_disconnected() never returns True

Under Uvicorn with --workers 1 and no --http httptools, disconnect detection can lag. Run with uvicorn app:app --http httptools --loop uvloop. Also verify you are actually awaiting the check inside the loop, not before it.

Usage block never arrives

You forgot stream_options={"include_usage": True}, or you are on an older OpenAI SDK version. Requires openai>=1.14.

Double-writes to the message row

You are still POSTing the assembled response from the client after the stream closes. Delete that code path. The server-side finally block is the single writer.

Tokens arrive in bursts, not smoothly

Usually TLS record buffering or gzip. Disable compression on the SSE route — Content-Encoding: identity. SSE payloads are small; compression buys you nothing and costs you smoothness.

What this pattern is bad at

Honest tradeoffs:

• Not great for very long-running generations (agent loops, tool calls that take minutes). SSE over a single HTTP connection is fragile past a few minutes. For those, switch to WebSockets or a job queue with polling.
• Horizontal scaling requires sticky sessions or a pub/sub layer. If your load balancer routes the resume request to a different pod than the original stream, you need Redis pub/sub or similar to coordinate. For most seed-stage SaaS at single-digit-thousand DAU, one pod is fine.
• Client-side cancellation is best-effort. AbortController on the browser closes the TCP connection, but the OpenAI API call on the server keeps running until the next chunk arrives and you break the loop. You will pay for a few extra tokens after cancel.

The pattern optimizes for the 95% case: a chat UI where users send a prompt, watch tokens stream in, and occasionally close the tab. It does that reliably and keeps your billing table honest.

Frequently Asked Questions

Should I use Server-Sent Events or WebSockets for LLM streaming?

SSE for chat-style streaming where the server pushes tokens one-way. It works over standard HTTP, survives proxies with the right headers, and reconnects cleanly. Use WebSockets when you need bidirectional messaging mid-stream — for example, an agent that asks the user for clarification while running, or a collaborative canvas.

How do I stream LLM responses in a Next.js app with the App Router?

The same server-side generator pattern applies. Use a Route Handler returning a Response with a ReadableStream body, and read it in the client with fetch plus a stream reader. Vercel AI SDK wraps this nicely, but be aware it defaults to accumulating on the client — you still need a server-side write in the stream itself if you want disconnect-safe persistence.

How do I bill accurately when a user cancels mid-stream?

Two options. Either accept that OpenAI only reports usage in the final chunk and bill zero for interrupted streams (users benefit slightly, you lose a small amount), or count buffered tokens with tiktoken in your finally block before writing. The second is more accurate but requires keeping tokenizer versions in sync with the model.

Can I use this pattern with Anthropic Claude or open-source models?

Yes. Anthropic's Python SDK exposes stream=True with an equivalent async iterator. For self-hosted models via vLLM or TGI, both expose OpenAI-compatible streaming endpoints, so the FastAPI code is unchanged — swap the base URL. The disconnect handling and DB write pattern is model-agnostic.

We need help implementing this in a production SaaS with auth, multi-tenancy, and audit logs. Where do we start?

For a personalized assessment of the streaming architecture, billing integration, and multi-tenant considerations, talk to the CodeNicely AI studio team. We have shipped this pattern into several production SaaS products, including cases where token accounting feeds directly into invoicing.