Home › Blog › Startups › Saas › Stream LLM Responses to a React Frontend Without Melting
SaaS technology
Startups SaaS July 7, 2026 • 11 min read

Stream LLM Responses to a React Frontend Without Melting

For: A mid-level full-stack engineer at a seed-stage SaaS startup who has a working ChatGPT-style feature that waits for the full LLM response before rendering — users are complaining it feels broken during the 4–8 second silence — and every tutorial they find either uses a third-party SDK abstraction that hides how streaming actually works or breaks when they try to add auth headers and parse partial JSON safely

If your LLM feature blocks for 4–8 seconds before rendering, you are waiting on the full completion instead of streaming. The fix is a Node proxy that forwards OpenAI's stream: true response as Server-Sent Events, plus a React consumer that reads fetch's ReadableStream as a state machine with explicit done, error, and abort transitions. The tutorial below is the minimum runnable version that survives auth headers, mid-stream failures, and slow 4G.

Most tutorials skip the parts that actually break in production: what happens when the socket dies at token 47, how to inject a bearer token (EventSource cannot), and how to parse structured JSON that arrives split across chunks. We will handle all three.

Why your current implementation feels broken

You almost certainly have this pattern somewhere:

const res = await fetch('/api/chat', { method: 'POST', body });
const { text } = await res.json();
setMessage(text);

The server is waiting on OpenAI to finish generating. OpenAI is generating at ~30–80 tokens/sec. A 400-token answer means multiple seconds of silence. Users assume it is broken and click again, which is how you get duplicate requests and duplicate charges.

The insight most tutorials miss: opening a stream is easy. Consuming it correctly under real network conditions is not. EventSource looks simpler than fetch + ReadableStream, until you need to send a POST body or an Authorization header — neither of which EventSource supports. And a naive for await (const chunk of reader) loop will silently swallow errors, double-append tokens on retry, or leak the connection if the component unmounts mid-stream.

Prerequisites

Step 1: Set up the Node proxy

Never call OpenAI from the browser — your key will get scraped in hours. Create a tiny Express server:

mkdir llm-stream && cd llm-stream
npm init -y
npm i express cors
node --version  # should be 18+

Create server.js:

import express from 'express';
import cors from 'cors';

const app = express();
app.use(cors());
app.use(express.json());

app.post('/api/chat', async (req, res) => {
  const { messages } = req.body;

  const upstream = await fetch('https://api.openai.com/v1/chat/completions', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'Authorization': `Bearer ${process.env.OPENAI_API_KEY}`,
    },
    body: JSON.stringify({
      model: 'gpt-4o-mini',
      messages,
      stream: true,
    }),
  });

  if (!upstream.ok) {
    const err = await upstream.text();
    return res.status(upstream.status).json({ error: err });
  }

  res.setHeader('Content-Type', 'text/event-stream');
  res.setHeader('Cache-Control', 'no-cache, no-transform');
  res.setHeader('Connection', 'keep-alive');
  res.setHeader('X-Accel-Buffering', 'no'); // disable nginx buffering

  const reader = upstream.body.getReader();
  const decoder = new TextDecoder();

  req.on('close', () => reader.cancel());

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    res.write(decoder.decode(value, { stream: true }));
  }
  res.end();
});

app.listen(3001, () => console.log('Proxy on :3001'));

Note three things: X-Accel-Buffering: no disables nginx's default buffering (this alone has burned a lot of teams), req.on('close') propagates client disconnects upstream so you stop paying for tokens the user will never see, and we forward OpenAI's SSE format as-is instead of re-parsing on the server.

Expected output: Proxy on :3001. If you curl -N -X POST http://localhost:3001/api/chat -H 'Content-Type: application/json' -d '{"messages":[{"role":"user","content":"count to 5"}]}', you should see data: {...} lines streaming in one at a time, ending with data: [DONE].

Step 2: Understand what you are consuming

OpenAI streams SSE frames that look like this:

data: {"choices":[{"delta":{"content":"Hello"}}]}

data: {"choices":[{"delta":{"content":" world"}}]}

data: [DONE]

Three edge cases that will bite you:

Step 3: The React hook — state machine, not a loop

Here is the consumer. It uses fetch (not EventSource) so we can send POST bodies and auth headers:

// useLLMStream.js
import { useState, useRef, useCallback } from 'react';

type Status = 'idle' | 'streaming' | 'done' | 'error' | 'aborted';

export function useLLMStream() {
  const [text, setText] = useState('');
  const [status, setStatus] = useState<Status>('idle');
  const [error, setError] = useState<string | null>(null);
  const abortRef = useRef<AbortController | null>(null);

  const send = useCallback(async (messages, authToken) => {
    // Cancel any in-flight stream before starting a new one
    abortRef.current?.abort();
    const controller = new AbortController();
    abortRef.current = controller;

    setText('');
    setError(null);
    setStatus('streaming');

    try {
      const res = await fetch('http://localhost:3001/api/chat', {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json',
          'Authorization': `Bearer ${authToken}`,
        },
        body: JSON.stringify({ messages }),
        signal: controller.signal,
      });

      if (!res.ok) {
        const body = await res.text();
        throw new Error(`HTTP ${res.status}: ${body}`);
      }
      if (!res.body) throw new Error('No response body');

      const reader = res.body.getReader();
      const decoder = new TextDecoder();
      let buffer = '';

      while (true) {
        const { done, value } = await reader.read();
        if (done) break;

        buffer += decoder.decode(value, { stream: true });
        const lines = buffer.split('\n');
        buffer = lines.pop() ?? ''; // keep partial line for next chunk

        for (const line of lines) {
          const trimmed = line.trim();
          if (!trimmed.startsWith('data:')) continue;
          const payload = trimmed.slice(5).trim();
          if (payload === '[DONE]') {
            setStatus('done');
            return;
          }
          try {
            const json = JSON.parse(payload);
            const delta = json.choices?.[0]?.delta?.content;
            if (delta) setText(prev => prev + delta);
          } catch {
            // Partial JSON — safe to skip, next chunk will complete it
          }
        }
      }
      setStatus('done');
    } catch (err) {
      if (err.name === 'AbortError') {
        setStatus('aborted');
      } else {
        setError(err.message);
        setStatus('error');
      }
    }
  }, []);

  const abort = useCallback(() => abortRef.current?.abort(), []);

  return { text, status, error, send, abort };
}

The important bits:

Step 4: The component

function Chat() {
  const { text, status, error, send, abort } = useLLMStream();
  const [input, setInput] = useState('');

  return (
    <div>
      <textarea value={input} onChange={e => setInput(e.target.value)} />
      <button
        onClick={() => send([{ role: 'user', content: input }], 'YOUR_JWT')}
        disabled={status === 'streaming'}
      >Send</button>
      {status === 'streaming' && <button onClick={abort}>Stop</button>}
      <pre>{text}</pre>
      {status === 'streaming' && <span>▋</span>}
      {status === 'error' && <div style={{color:'red'}}>{error}</div>}
    </div>
  );
}

Expected output: Type a prompt, click Send. Tokens appear within ~200–400ms and stream in continuously. The Stop button aborts cleanly; no error banner appears.

Step 5: Handle structured JSON responses

Plain text streaming is the easy case. If you are asking the model for JSON (function calling, structured output), the JSON itself arrives in fragments. You cannot JSON.parse partial JSON.

Two workable approaches:

  1. Accumulate then parse. Buffer the full content string, only parse on [DONE]. Simple. Users see nothing until completion — defeats streaming for JSON.
  2. Incremental parsing with a permissive parser. Libraries like partial-json can parse {"name": "Ali and return { name: 'Ali' }. Good for rendering as fields arrive.
import { parse, Allow } from 'partial-json';

let accumulated = '';
// inside your delta handler:
accumulated += delta;
try {
  const partial = parse(accumulated, Allow.ALL);
  setParsedObject(partial); // render fields that exist so far
} catch {
  // not enough yet
}

Step 6: Backpressure and the double-render trap

On slow networks (throttled 4G in DevTools is a good test), chunks arrive irregularly. Two failure modes to watch for:

Step 7: Deploy behind a proxy that does not buffer

The number one production surprise: your local setup works, you deploy behind nginx or Cloudflare, and streaming turns back into blocking. Fixes:

Common errors

Response arrives all at once instead of streaming

A proxy is buffering. Check for proxy_buffering, Cloudflare, or a CDN in front of your API. Test directly with curl -N against your origin — if that streams and the browser doesn't, the culprit is between them.

SyntaxError: Unexpected end of JSON input

You are calling JSON.parse on a partial line. Wrap the parse in a try/catch and skip on failure. The next chunk will complete the JSON.

Tokens double up after network hiccup

You have retry logic that does not reset the accumulated text. On retry, either start fresh or resume from a known offset — never both.

EventSource does not work with my auth

Correct. EventSource cannot send custom headers or POST bodies. Use fetch + ReadableStream as shown above. This is why the tutorial does not use EventSource.

Stream stalls on the first token in production

Likely gzip/brotli compression buffering the response. Disable compression on the streaming route, or ensure your proxy is set to flush on write.

Component unmounts but the request keeps running

You are not calling abort() in a cleanup. Add useEffect(() => () => abort(), [abort]) in the parent, or track the controller in a ref and abort on unmount.

What this approach is bad at

Honest tradeoffs:

For most seed-stage SaaS teams building a ChatGPT-style feature, none of these are day-one problems. Ship the state-machine consumer, disable proxy buffering, and revisit reconnection when you have paying users complaining about it. Teams working on more involved LLM product surfaces — RAG pipelines, agent workflows, multi-tenant inference — usually end up needing a purpose-built streaming layer, which is the kind of work we tackle in our AI studio.

Frequently Asked Questions

Should I use Server-Sent Events or WebSockets for LLM streaming?

SSE for anything one-directional (server pushes tokens to client). It runs over plain HTTP, works through most proxies, and reconnects on its own if you use the EventSource API. WebSockets only make sense if you also need the client pushing structured messages mid-stream — for a chat UI, you do not.

Why not just use the OpenAI Node SDK's streaming helper?

The SDK is fine for the server-to-OpenAI hop. The problem is that many tutorials pipe the SDK's async iterator directly into a WebSocket or SSE handler without buffering, and it breaks the moment you add middleware, auth, or a CDN. Owning the byte-level forwarding as shown above makes failure modes visible instead of magical.

How do I handle authentication for the streaming endpoint?

Send a bearer token in the Authorization header on the fetch call — this is exactly why we avoid EventSource, which cannot send custom headers. Validate the token in your Node proxy before opening the upstream connection to OpenAI. For session tokens that rotate, refresh before the stream starts, not during.

Can I stream from OpenAI directly in a Next.js API route?

Yes, but only in an Edge Runtime route or a Node runtime route that returns a Response with a ReadableStream body. The default serverless runtime on some platforms buffers the response. Check your platform's docs for "streaming responses" specifically — it is not the default everywhere.

How much does streaming cost extra compared to non-streaming?

Token usage is identical. What changes is connection duration and infrastructure behavior (open sockets, proxy timeouts, serverless invocation billing). For a specific assessment of streaming architecture for your product, talk to CodeNicely for a personalized review.

Keep reading

SaaS

Microservices vs. Modular Monolith: Pick the Right Architecture

Your monolith is slow to deploy and teams are stepping on each other. Before you split into microservices, here's how to tell if your real problem is coupling, pipeline design, or actual scale — and which architecture solves which.

 SaaS

AI Feature Flags Cheatsheet: What to Gate, Rollback, and Monitor

Standard feature flags gate users. AI features need a second layer that gates model behavior — version, prompt, embedding schema, output quality. Here's the reference cheatsheet.

 SaaS

Celery vs. Temporal: Pick the Right Workflow Engine for AI Jobs

Celery loses in-flight state when a worker dies mid-chain — and no amount of logging fixes that. Here's an honest head-to-head with Temporal for teams running multi-step LLM and document-parsing pipelines in production.

Found this useful? CodeNicely publishes engineering and product playbooks weekly. Browse the archive or tell us what you're building.