5 Mistakes Teams Make When Embedding AI into a Live E-Commerce Operation

For: COO or Head of Operations at a mid-size e-commerce or e-pharmacy business that has approved an AI initiative to automate order routing, inventory, or fulfillment — and is 60–90 days into rollout watching it underperform against the pilot results

If your AI order routing, inventory, or fulfillment model beat every benchmark in the pilot and is now underperforming 60 days into production, the model is probably fine. The integration around it is not. In almost every underperforming rollout we've been asked to rescue, the root cause is the same: the training data was pulled from the clean, completed-order history in the production database, which by definition excludes every exception, cancellation, split shipment, address correction, and manual override that operations staff handle every day. The model was never taught the cases that now dominate its production workload.

The rest of this post walks through the five mistakes we see most often when teams embed AI into a live e-commerce or e-pharmacy operation, what each one looks like in production, and how to recover without ripping the model out.

Mistake 1: Training on completed orders and calling it representative

What causes it: The data science team asks operations for "a year of order data." Ops exports the orders table. That table contains the orders that made it cleanly through the funnel — paid, picked, shipped, delivered. It does not contain the 8–15% of orders that got stuck: the address that failed verification, the SKU that was out of stock at the assigned warehouse, the prescription that needed pharmacist review, the COD order the customer refused at the door, the split shipment that had to be reassembled.

Symptom in production: The model performs beautifully on the 85% of orders that would have flowed through with or without it. The remaining 15% land in a manual exception queue that grows faster than ops can clear it. Leadership sees the queue and assumes the model is wrong. It isn't — it just has no policy for cases it never saw.

How to recover: Stop retraining. Instead, pull the exception logs, the CRM tickets, the WhatsApp/Slack threads where CX escalated problem orders, and the manual override records from your OMS or WMS. This is where the operational knowledge lives. Label those cases with the outcome ops actually chose, and retrain with exceptions weighted at their true production frequency, not their historical frequency in the clean table. In our HealthPotli work — an e-pharmacy platform where every order has to clear drug interaction, prescription validity, and substitution logic — the exception data was the training signal that made the AI useful. The completed-order data was almost noise.

Mistake 2: Treating the AI decision as the endpoint instead of a step in a workflow

What causes it: The pilot was framed as "can the model predict the right warehouse / the right courier / the right replenishment quantity." It could. So the team wired the model output directly into the OMS and moved on. Nobody designed what happens when the model returns low confidence, when downstream systems reject the decision, or when the decision is right but the state of the world changed between prediction and execution (a courier goes down, a SKU gets flagged, a pin code goes into a lockdown zone).

Symptom in production: Orders get routed to a warehouse that is technically optimal but out of picking capacity. Replenishment orders get placed for SKUs that a buyer manually delisted yesterday. The model is right on paper and wrong in the aisle.

How to recover: Wrap the model in a decision layer, not an API call. That layer needs three things: a confidence threshold that routes low-confidence cases to a human queue with the model's top-3 suggestions pre-populated, a real-time constraint check against current warehouse capacity and SKU status before the decision is committed, and an override log that captures every time ops overrode the model and why. That override log becomes your next training set. This is the pattern we used on Vahak's route and load matching — the model proposes, the constraint layer disposes, and every human override feeds back into the next model version.

Mistake 3: No shadow-mode period before the model started making live decisions

What causes it: Pressure to show ROI. The pilot worked, leadership approved, and the team flipped the model to live decisions on day one of rollout. There was no window where the model's predictions were logged and compared against what ops actually did, without the model being in control.

Symptom in production: You have no baseline. When the queue starts backing up, you cannot answer the only question that matters: is the model worse than the humans it replaced, or is it better but exposing problems ops used to hide with manual workarounds? Without a shadow-mode log, every discussion becomes anecdotal — the loudest CX manager wins.

How to recover: Even after go-live, you can retrofit this. Turn on decision logging for both the model output and the final human-or-system action taken. Compute the delta weekly by order type, region, SKU category, and value band. You will almost always find that the model is beating humans on the boring 80% and losing on a specific, identifiable slice — high-value orders, cross-border shipments, orders with special handling flags. Now you know exactly where to intervene, and you have data to defend the rollout to a nervous CEO.

Mistake 4: Ignoring the feedback loop between AI decisions and staff behavior

What causes it: When a warehouse supervisor doesn't trust a routing decision, they don't file a ticket. They open the OMS, change the assignment, and move on. When a buyer thinks the replenishment quantity is wrong, they edit the PO before it goes to the vendor. These overrides are invisible to the AI team because they happen inside tools that were never instrumented for it.

Symptom in production: The dashboards say the model made 40,000 decisions this month with 92% "acceptance." On the floor, staff are silently overriding 20% of them. The model never learns it was wrong. Worse, the staff learn that the model is unreliable and start overriding cases where it was actually right — the trust is gone and there's no mechanism to rebuild it.

How to recover: Instrument the override. Every edit to a model-generated field in the OMS, WMS, or procurement tool should be captured with the old value, the new value, the user, and — this is the hard part — a required reason code from a short, tightly controlled list ("warehouse capacity," "customer preference," "SKU issue," "other + free text"). Then publish an override report back to the same team every week. When staff see that their overrides are visible and are being used to improve the model, two things happen: override rates drop on the cases where the model was right, and the free-text "other" bucket becomes the richest source of edge cases you have. This is the loop that makes operationalized AI compound over time instead of decay.

Mistake 5: One model for everything instead of a portfolio

What causes it: A single team ships a single "AI for fulfillment" model that tries to handle order routing, courier selection, split-shipment logic, and returns triage. It performs acceptably on the average order and badly on every specific segment. Because it's one model, you cannot fix one segment without risking the others.

Symptom in production: Every improvement to prescription-order handling breaks something in the OTC flow. Every tweak to metro-city routing degrades tier-2 performance. The team stops shipping changes because the blast radius is too big.

How to recover: Break the monolith into a small set of narrower models with a clear routing layer on top. In e-pharmacy, that usually looks like separate models for prescription validation, substitution suggestions, warehouse assignment, and last-mile courier selection — each trained on its own data, each evaluated on its own metrics, each deployable independently. The routing layer decides which model to call based on order attributes. This is slower to build up-front and dramatically faster to improve after month three. It is also the only structure we've seen where the ops team can point at a specific model and say "this one is the problem" — which is a prerequisite for fixing anything.

What a healthy AI-in-operations rollout actually looks like at 90 days

If you're doing this well, at day 90 you should be able to answer, in one meeting, all of the following: what percentage of decisions the model is making autonomously versus routing to humans, what the override rate is by segment and by reason code, which specific slice of orders the model is worse than humans on, what the shadow-mode delta was in the last two weeks, and which model in the portfolio is scheduled for the next retrain and why. If you can't answer those, the problem isn't the model. It's that embedding AI into business operations was treated as a deployment instead of an operating system change.

How CodeNicely can help

Most of what's in this post came from cleaning up rollouts that were shipped before the integration layer was designed. The closest analog to a typical e-commerce or e-pharmacy rescue is our work on HealthPotli — an e-pharmacy operation where AI had to sit inside a live order flow with prescription validation, drug interaction checks, substitution logic, and last-mile constraints all interacting on every single order. The engagement was less about model accuracy and more about building the decision layer, exception queues, override instrumentation, and retraining loop around the model so ops could actually run it. If your current rollout is stuck in the pattern this post describes — pilot worked, production is worse, exception queue is growing, staff are routing around the system — that's the specific shape of problem we handle. We also work across the broader digital transformation stack (legacy OMS/WMS modernization, custom middleware, automation) because AI in operations almost never fails in isolation. You keep full IP ownership and there is no vendor lock-in on the models, the pipelines, or the integration code.

Frequently Asked Questions

How do I know if my AI model is underperforming because of accuracy or because of integration?

Run a two-week decision log where you capture the model's prediction, the final action taken, and any override reason. If the model is right and the final action still went sideways, it's an integration or constraint problem. If the model is wrong on a specific segment more than 15% of the time, it's a training data problem — almost always because exceptions weren't in the training set.

Should we pause the AI rollout while we fix the training data?

Usually no. Put the model into a hybrid mode where high-confidence decisions run autonomously and everything below a threshold routes to humans with the model's suggestion attached. Fix the training data in parallel using the exception logs and override reasons you're now capturing. Pausing entirely tends to kill organizational trust in the initiative.

How much of my exception data do I need to retrain effectively?

Enough to make each exception class statistically meaningful — usually a few hundred labeled examples per exception type is a working starting point, though it depends on the model architecture. What matters more than raw volume is that the exceptions are labeled with the outcome ops actually chose and the reason, not just flagged as "exception."

Can we reuse the pilot model or do we need to start over?

Almost always reuse. The pilot model is rarely the problem. What needs to be built is the decision layer around it — confidence routing, constraint checks, override instrumentation, and the retraining pipeline. Starting over on the model usually means repeating the same mistake with a different algorithm.

What does a rescue engagement for a stuck AI rollout look like with CodeNicely?

It typically starts with a two-to-three week diagnostic: we instrument the current system, capture the decision and override data that's missing, and map where the model is actually failing versus where the integration is failing. From there the scope depends on what we find. For a personalized assessment of your specific rollout, talk to our team.

If your team is 60–90 days into an AI rollout and the numbers are moving the wrong way, the fix is almost never a better model. It's a better decision layer, honest exception data, and a feedback loop your ops team actually uses. Get those three right and the model you already have will start earning its keep.