Your AI Feature Isn't Slow. Your Data Contract Is.

For: A Series B SaaS CTO whose newly shipped AI feature has 3–6 second response times in production and whose engineering team is pressure-testing the model and inference layer — convinced the bottleneck is AI when it almost certainly isn't

If your AI feature is responding in 3 to 6 seconds and your team is benchmarking inference providers, you're probably about to spend two sprints optimizing the wrong layer. The bottleneck almost certainly isn't the model. It's the implicit, undocumented data contract between your product database and the AI layer — the assumptions about field shape, freshness, nullability, and ownership that nobody wrote down because nobody had to until an LLM started consuming them.

This is the unglamorous truth of AI product engineering in 2024: prompt engineering and model selection are the parts that get blog posts. Data contracts are the parts that decide whether your feature works.

The thesis

The majority of production AI feature failures that look like model problems are data contract failures. Specifically: mismatches in field nullability, staleness tolerance, schema drift, and ownership boundaries that the AI layer assumes are stable and the upstream services treat as soft suggestions. Tuning temperature, switching from GPT-4o to Claude, or moving to a smaller model on dedicated hardware will not fix any of this. It will just make the broken thing fail faster.

Here's the test: instrument your AI endpoint and break the latency down by phase. In almost every slow AI feature I've seen audited, the actual model call is somewhere between 600ms and 1.4 seconds. The remaining two-to-five seconds are spent fetching, joining, normalizing, retrying, and waiting on data that the team assumed was sitting in a row, ready to go.

Why this happens — and why your team will resist hearing it

AI features get built by your strongest engineers. Strong engineers reach for the most interesting problem in the room, and the model is more interesting than the join. So the team writes a clean prompt, picks a model, builds a retry layer, and ships. The data fetch is treated as plumbing — a function called getUserContext() that returns whatever it returns.

Then production traffic hits, and three things happen at once:

• Fields the prompt assumes are populated are sometimes null, because the upstream service treats them as optional. The AI layer either errors, falls back to a slower path, or quietly produces worse output that someone files a bug about next week.
• The data the prompt assumes is fresh is hours stale, because it lives in a read replica or a denormalized cache nobody told the AI team about. The team adds a real-time fetch as a fix. Latency doubles.
• Schema changes upstream — a renamed column, a new enum value, a JSON blob that grew a nested field — and the AI layer's silent assumptions break. Nobody notices for a week because the model is generous about malformed input.

None of this shows up in a model benchmark. All of it shows up in p95 latency and accuracy degradation in production.

Three concrete patterns I see repeatedly

1. The N+1 prompt

An AI feature for a B2B SaaS app needs to summarize a customer's recent activity. The engineer writes a prompt that takes a list of events. The fetch function pulls events one at a time from an internal API because that's what the API exposes. Twenty events, twenty round-trips, each 80ms. That's 1.6 seconds before the model sees a token. The team blames the model and tries streaming. Streaming doesn't help, because streaming starts after the prompt is assembled.

The fix is a batch endpoint or a denormalized projection. The fix is not a different LLM.

2. The freshness mismatch

An AI assistant in a fintech product references the user's current balance. The product database is the source of truth, but the analytics warehouse — which the AI layer was wired into for convenience during the prototype — lags by 15 to 90 minutes. In demo, it works. In production, users ask about a transaction they made 10 minutes ago and the AI confidently tells them it didn't happen. The team interprets this as a hallucination problem and starts adding guardrails. The actual problem is that the contract said "current balance" and the pipeline delivered "balance as of last warehouse sync." In domains like lending and KYC, this kind of staleness mismatch isn't a UX issue — it's a compliance issue.

3. The nullability tax

A healthcare-adjacent AI feature assumes every user has a complete profile: date of birth, prescriptions, allergies. In reality, 30% of users have partial profiles because onboarding is progressive. The prompt template handles missing fields by inserting the string "unknown," which the model interprets as a meaningful signal and starts producing oddly hedged output. The team tunes the prompt for weeks. The actual fix is upstream: enforce a minimum-viable-context check before the AI route is even callable, and route incomplete profiles to a deterministic fallback. We saw a version of this in the HealthPotli drug interaction work — the AI's quality was bounded entirely by what the data layer could guarantee about the input.

What a real data contract looks like

A data contract for an AI feature isn't a Confluence doc. It's an enforceable agreement, ideally codified, that specifies:

• Shape: exact field names, types, and which are non-nullable for the AI path. Validated at the edge, not inside the prompt.
• Freshness SLO: maximum acceptable staleness per field. "User balance: <5s. User preferences: <60s. Product catalog: <1h." If the source can't meet it, the AI feature should know and degrade explicitly.
• Ownership: which team owns the producer, who they page when the contract breaks, and what the deprecation policy is for schema changes.
• Failure mode: what the AI layer does when the contract is violated. Fall back, refuse, or proceed with a labeled degraded response. Never "silently inject a string and hope."

The teams I see ship reliable AI features treat the data contract as a first-class artifact, often before the prompt is written. The teams stuck on 4-second p95s wrote the prompt first.

The honest counter-argument

The strongest pushback to this thesis is that sometimes the model genuinely is slow. A 200K-token context window with a frontier model and no caching will take seconds no matter what you do upstream. RAG over a large vector store with poor index hygiene will dominate the latency budget. If you're doing agentic workflows with multi-step reasoning, model calls are the floor, and the floor is high.

Fair. But here's the heuristic: if your feature does a single model call with a prompt under 8K tokens and you're seeing >2 seconds total latency, the data layer is the suspect. If you're doing multi-turn agentic work or large-context retrieval, the model layer is a legitimate suspect — but you should still instrument the data fetch first, because the cheapest wins are almost always there.

Either way, the diagnostic order is the same: measure each phase before you optimize any of them. The number of teams that skip this and go straight to inference shopping is, frankly, what this essay is about.

What to do Monday morning

1. Instrument the AI endpoint by phase. Auth, data fetch (per source), prompt assembly, model call, post-processing. Log each. You will be surprised within a day.
2. Find the slowest non-model phase and ask what contract it's enforcing. Usually the answer is "none, we just call this function." That's the bug.
3. Write the contract for that one phase. Shape, freshness, ownership, failure mode. Codify it — Pydantic, Zod, JSON Schema, whatever your stack uses. Validate at the boundary.
4. Add a fallback path for contract violations. Not a try/except that swallows. An explicit degraded response the user can recognize.
5. Only after that, look at the model. Now your benchmarks will mean something, because you'll be measuring the model, not the model plus four seconds of fragile plumbing.

The reason ML pipeline latency feels mysterious is because most teams treat the AI layer as a black box and the data layer as solved. It's the inverse. The model is the most predictable, well-benchmarked component in your stack. Your data is the wild thing. Treat it that way and most of your slow AI integration problems stop being AI problems at all.

Frequently Asked Questions

How do I know if my slow AI feature is a model problem or a data problem?

Instrument latency by phase: data fetch, prompt assembly, model call, post-processing. If the model call is under 1.5 seconds and total latency is 3+ seconds, the bottleneck is upstream of the model. For single-call features with prompts under 8K tokens, the data layer is almost always the larger contributor.

What is a data contract in the context of AI features?

An enforceable agreement between the producer of data and the AI layer that consumes it, specifying field shape, nullability, freshness SLOs, ownership, and failure modes. Unlike a documentation page, it's codified and validated at runtime — typically with schema validation libraries — so contract violations fail loudly instead of silently corrupting prompt input.

Will switching to a faster inference provider fix my latency?

Sometimes, but rarely as much as teams expect. If your model call is 1 second and your total latency is 5 seconds, moving to a provider that's 30% faster saves you 300ms on a 5-second response. The leverage is in the other 4 seconds. Optimize the data path first, then revisit inference.

How does schema drift affect AI features differently than traditional services?

Traditional services fail loudly on schema changes — a missing field throws an exception. LLMs are generous about malformed input and will often produce plausible-looking output from broken data, which means schema drift causes silent quality regressions instead of obvious errors. This makes drift especially dangerous in AI paths, and is why explicit contract validation at the boundary matters more, not less.

We're a Series B startup — should we build all this contract infrastructure ourselves?

Most of it is achievable with existing tools — Pydantic or Zod for shape validation, your observability stack for phase-level latency, and a clear ownership model for upstream services. The harder part is the organizational discipline of treating data contracts as first-class artifacts. If you'd like a second set of eyes on a specific AI feature that's underperforming, contact CodeNicely for a personalized assessment.