How to Migrate a Live E-Commerce Catalog to AI Search

For: Head of Product at a mid-market e-commerce company (50K–500K SKUs, $10M–$100M GMV) whose keyword search is returning irrelevant results at scale and whose engineering team has prototyped a vector/semantic search replacement but has no safe rollout plan that won't tank conversion during the cutover

Migrate AI search behind a shadow-traffic and percentage-based rollout, not a flag flip — and treat catalog data cleanup as part of model evaluation, not a separate workstream. Your offline benchmarks are almost certainly optimistic because the SKUs with sparse descriptions also tend to be the SKUs your eval set under-represents. If you skip that step, you'll ship confidently and discover the failure mode three weeks in, when long-tail and seasonal queries quietly stop converting.

This playbook is for a Head of Product at a mid-market retailer with 50K–500K SKUs whose engineering team has a working vector search prototype but no safe rollout plan. It assumes you already have an embedding model picked (open or hosted), a vector store stood up, and an offline relevance score that looks better than your current Elasticsearch/Solr/Algolia setup. The job now is getting it into production without losing money.

The situation this applies to

• You run keyword search today (Elasticsearch, OpenSearch, Solr, Algolia, or a vendor SaaS) and it's degrading as the catalog grows.
• Your team has prototyped semantic or hybrid search using embeddings + a vector DB (pgvector, Pinecone, Weaviate, Qdrant, Vespa, OpenSearch k-NN).
• Offline eval (NDCG@10, recall@50, or human-rated relevance) looks meaningfully better than the incumbent.
• Search drives a large share of GMV — somewhere between 30% and 70% of sessions touch it — so a bad cutover shows up in the weekly board deck.
• Roughly 15–20% of your catalog has thin, inconsistent, or auto-generated product copy: marketplace seller listings, drop-ship feeds, seasonal SKUs, refurbished inventory.

If that's you, here's the order to do this in.

Step 1: Build the query set before you touch the model

Most teams evaluate semantic search on a sampled query log weighted by frequency. That's the wrong eval set. Frequent queries are the ones keyword search already handles fine — "nike air max 90 black size 10" works in any system. The queries that justify replacing keyword search are the ones that fail today: descriptive ("shoes for standing all day on concrete"), compositional ("red dress for outdoor fall wedding under knee"), and long-tail head terms with zero results.

Pull four buckets from the last 90 days:

1. Head queries with high conversion — your regression set. The new system must not lose ground here.
2. Head queries with low CTR or high null rate — your improvement set. This is where the win lives.
3. Long-tail descriptive queries (≥4 words, no brand token). Sample at least 500.
4. Zero-result queries. Sample 300+. These are pure upside.

For each bucket, get human relevance judgments on the top 10 results from both systems for ~200 queries. Use your merchandising team or a labeling vendor — not engineers, who will rationalize the model's choices.

Anti-pattern: Evaluating only on click-through data. CTR is biased toward what keyword search surfaced; you can't measure recall on results users never saw.

You'll know this step is done when you have a labeled eval set of 800–1,000 queries split across the four buckets, and you can compute NDCG@10 separately for each bucket. If your new system wins on aggregate but loses on bucket 1 (head/converting), do not ship.

Step 2: Audit the catalog against the eval set — and fix the bottom 20% first

Here's the non-obvious part. Your offline benchmark and your catalog data quality problem are the same problem. Embeddings on a product with a 12-word title and no description will land somewhere noisy in vector space. That product won't surface for queries it should match, and — worse — it'll surface for queries it shouldn't, because the embedding has nothing specific to anchor against.

Run this audit before any rollout:

• Compute description length, attribute completeness (brand, category, material, color, size, use case, audience), and image count for every SKU.
• Score each SKU 0–5 on "embed-ability." Anything with title-only or fewer than three structured attributes goes in the bottom bucket.
• Cross-reference the bottom bucket against last-90-day GMV. The intersection of "thin data" and "actually sells" is your priority queue.

Now enrich. Use an LLM (GPT-4-class or Claude) to generate structured attributes and expanded descriptions from titles, images (vision models), and category context. Have merchandising review a sample — don't auto-publish LLM output to a live PDP without spot-checks, because hallucinated specs are a returns and trust problem. For attributes you can derive deterministically (extracting color from an image, parsing size from a title), prefer that to LLM generation.

Anti-pattern: Re-running offline eval on the original catalog after enrichment and celebrating the lift. Of course it goes up. The honest comparison is: original system on enriched catalog vs. new system on enriched catalog. Otherwise you're crediting the model for work the data team did.

You'll know this step is done when the bottom-bucket SKUs that account for the top 5% of revenue have attribute completeness scores comparable to your median SKU, and your eval lift holds after re-running on the cleaned catalog.

Step 3: Run hybrid, not pure semantic

Pure vector search loses on exact-match queries. Always. If a user types a SKU code, a UPC, or a precise model number, BM25 will smoke any embedding model. The production answer is hybrid: run both retrievers in parallel and fuse with Reciprocal Rank Fusion (RRF) or a learned reranker.

Practical setup:

• Keep your existing BM25 / keyword index. Don't decommission it.
• Add a vector index (pgvector if you're on Postgres and your SKU count is under a few million; Qdrant, Weaviate, or OpenSearch k-NN at higher scale).
• Fuse with RRF (k=60 is a fine default) for v1. Move to a cross-encoder reranker (bge-reranker, Cohere Rerank) on the top 50–100 fused candidates once you have query latency budget.
• Add query routing: numeric SKUs, UPCs, and brand+exact-model patterns bypass semantic entirely and go straight to keyword.

Anti-pattern: Replacing keyword search. You're augmenting it. The teams that get into trouble are the ones who frame this as "deprecate Elasticsearch" instead of "add a semantic layer."

You'll know this step is done when your hybrid system wins on every query bucket from Step 1, including head/exact-match, and p95 latency is within 50ms of your current search.

Step 4: Shadow-traffic before you serve a single user

Before any user sees a semantic result, run the new system in shadow mode for at least two weeks. Every real query hits both systems. Users see the old results. You log both result sets.

What you're looking for:

• Result set divergence per query bucket. If 80%+ of head-query result sets are similar between old and new, your risk on that bucket is low.
• Catastrophic divergence cases. Queries where the new system returns wildly different products. Pull a sample of 100 and have merchandising rate them.
• Latency under real load. Offline benchmarks lie about tail latency. p99 is what kills you.
• Index freshness lag. When merchandising updates a product, how long until it's re-embedded and searchable? If your answer is "the next nightly batch," that's a launch blocker for any catalog with real velocity.

Anti-pattern: Skipping shadow because the prototype "worked fine in staging." Staging doesn't have your traffic shape, your bot traffic, your cache miss patterns, or your weird queries from affiliate links.

You'll know this step is done when you've reviewed catastrophic-divergence samples with merchandising and either fixed them or accepted them, and you have p50/p95/p99 latency numbers on real production query volume.

Step 5: Roll out by traffic slice, with conversion as the kill switch

Now you serve users. Not all of them. The rollout sequence I'd run:

1. 5% of traffic, randomly assigned, for 7–10 days. Measure search-to-PDP CTR, PDP-to-cart, search-to-purchase, null-result rate, search refinement rate (proxy for "users didn't find it"), and revenue per search session.
2. 25% for another week if metrics hold or improve. Watch for category-specific regressions — aggregate can look fine while "home goods" is down 8%.
3. 50%. By this point you should have enough data for category-level confidence intervals.
4. 100%.

Set automatic kill switches: if revenue per search session drops more than X% over a 24-hour window with statistical significance, route 100% back to the old system. Wire this into your existing experimentation platform (LaunchDarkly, Statsig, Optimizely, or homegrown) — don't ask engineers to flip it manually at 2am.

Segment your rollout analysis by:

• New vs. returning users (new users have no behavioral signal in either system; that's where semantic should win biggest).
• Query bucket from Step 1.
• Top-level category. Apparel and home goods behave very differently from electronics and consumables.
• Device. Mobile users type shorter, sloppier queries — semantic helps more here.

Anti-pattern: Watching only aggregate conversion. The category that's quietly tanking is the one with thin product data you didn't fix in Step 2.

You'll know this step is done when 100% of traffic is on the new system, kill switch hasn't fired, and revenue per search session is flat-to-up across every major category.

Step 6: Stand up the feedback loop before you walk away

The mistake teams make after a successful rollout is treating search as done. Semantic search degrades differently than keyword search — keyword fails loudly (zero results), semantic fails quietly (wrong but plausible results). You need monitoring built for that.

What to instrument:

• Null-result rate by query length and bucket. A spike means embedding drift or a catalog feed issue.
• Click-position distribution. If users are increasingly clicking position 5+ instead of 1–3, ranking is degrading.
• "No relevant result" signal. Add it as a UI affordance or infer it from refinement + zero-click sessions.
• Re-embedding pipeline health. When does a SKU edit propagate to the vector index? Alarm if lag exceeds your SLA.
• Periodic eval re-runs. Re-score your Step 1 eval set monthly. If NDCG drops, dig in.

Also: have a written rollback plan. Old index stays warm and indexed for at least 90 days post-cutover. The cost of keeping two indexes is trivial compared to the cost of needing the old one back and not having it.

You'll know this step is done when someone other than the original engineering team can read a dashboard and tell you whether search is healthy this week.

Failure modes I've seen

Shipping on offline eval alone. Team gets +18% NDCG@10 offline, ships to 100%, watches revenue per session drop 4% in week one. The eval set didn't represent the long tail, and the long tail is where conversion lives.

Embedding the title only. Cheaper, faster, and silently catastrophic on catalogs where the title is mostly a SKU code or a marketplace seller's keyword spam. Embed a structured concatenation: title + key attributes + a generated short description, in a consistent template.

Ignoring filter/facet interaction. Semantic search returns 200 candidates ranked by relevance. The user then applies a "size: M, in-stock" filter and 180 of them disappear. Your top result is now the 21st most relevant. Either over-fetch and filter, or constrain at retrieval time — but test the interaction explicitly.

Forgetting personalization. Your old keyword system probably had merchandising boosts, bestseller weighting, or stock-level penalties baked in. Pure semantic ranking ignores all of that. You'll need to layer business signals back on top of relevance score — usually as a reranking pass.

Re-embedding cost surprises. Some teams discover at month three that their hosted embedding API bill is meaningfully higher than the keyword search infra it replaced, because every catalog update triggers a re-embed and the catalog updates more than they thought. Model this in advance and consider self-hosting an open embedding model (bge-large, e5, nomic) for high-velocity catalogs.

How CodeNicely can help

The work above is roughly half ML, half platform engineering, half data quality (yes). Most product teams have one of those three and need the other two.

The closest analog in our case study list is Vahak — a logistics marketplace where search and matching quality directly drove transaction volume, and where we had to roll out ranking and routing changes on live traffic without breaking the marketplace for either side. The constraints are similar: high-stakes search, real users in session, no acceptable downtime, and a long tail of edge cases that only show up in production. We've also done LLM-based data enrichment at scale on HealthPotli, where messy, inconsistent pharmaceutical product data had to be normalized and made searchable without introducing safety errors — exactly the muscle this migration needs in Step 2.

If you want help structuring the rollout, building the shadow-traffic harness, or doing the catalog enrichment pass, talk to our AI studio team. We work in the codebase with your engineers, ship under your IP, and don't lock you into proprietary infra.

Frequently Asked Questions

Should we replace keyword search entirely or run hybrid?

Hybrid, almost always. Pure semantic search loses badly on exact-match queries — SKU codes, UPCs, specific model numbers — where BM25 is unbeatable. The production pattern is to run both retrievers and fuse them with RRF or a learned reranker, plus a query router that sends obvious exact-match patterns straight to keyword.

How long should we shadow-test before serving real users?

At least two full weeks of real production traffic, and longer if your business has weekly seasonality (most retail does). You're looking for catastrophic divergence cases, tail latency under load, and re-indexing lag — none of which show up in staging. The right answer depends on your traffic volume and catalog velocity; contact CodeNicely for a personalized assessment if you want help sizing it.

What's the best vector database for an e-commerce catalog?

For catalogs under a few million SKUs and teams already on Postgres, pgvector is usually enough and avoids adding infrastructure. Above that, or when you need advanced filtering and hybrid scoring natively, Qdrant, Weaviate, and OpenSearch k-NN are all solid. Vespa is the strongest option at very large scale but has a steeper operational curve.

How do we handle products with very thin descriptions?

Enrich before you embed. Use LLMs to generate structured attributes and expanded descriptions from titles, images (with vision models), and category context, then have merchandising review a sample before publishing. Prioritize the intersection of "thin data" and "actually sells" — that's where bad recall costs you revenue.

What metrics should trigger an automatic rollback?

Revenue per search session is the cleanest single metric — it captures both relevance (do users click?) and intent match (do they buy?). Set a statistically significant drop over a 24-hour window as your trip wire, segmented by major category so a regression in one vertical doesn't get masked by aggregate. Search-to-purchase rate and null-result rate are useful secondary alarms.