Microservices vs. Monolith for Your First AI Feature

For: A CTO at a Series A B2B SaaS company whose core product is a Django or Rails monolith — now shipping their first AI feature and unsure whether to bolt it onto the monolith or extract a separate ML service, with an engineering team of 4–8 people who have never operated a distributed inference system

Put the first AI feature in your monolith unless the model is retrained on a cadence independent of your application release cycle. That single test — independent retraining cadence — is the only one that reliably earns the operational cost of a separate service for a team of 4–8 engineers. Everything else (latency, language fit, GPU isolation) is a tiebreaker, not a deciding factor.

The rest of this post is the long version of that answer: why most microservices-vs-monolith advice for ML is wrong for your stage, the five axes that actually matter, and a clean rule for which situation maps to which choice.

The decision, stated crisply

You have a working Django or Rails monolith. You are about to ship your first AI feature — could be a classifier, an LLM call wrapped in retrieval, a recommendation model, a fine-tuned embedding lookup. You have to choose:

• Option A — In-monolith. Inference code lives in the same repo, deployed in the same pipeline, called as a function or a module. If you use a hosted model (OpenAI, Anthropic, Bedrock), the API client lives next to your business logic. If you self-host, the model loads into your app process or a sidecar on the same host.
• Option B — Separate service. A new repo, separate deploy pipeline, its own runtime (often Python + FastAPI even if the monolith is Rails), called over HTTP or gRPC. Owns its own model artifacts, its own observability, its own on-call surface.

Most architecture writing skips this framing and jumps straight to scaling diagrams. That is the wrong altitude. At Series A, with 4–8 engineers and zero prior experience operating distributed inference, the question is not what scales best. It is what fails least and ships fastest.

The non-obvious insight: split on retraining cadence, not on the model

Here is the mental model that cuts through every other consideration.

A service boundary is not free. It costs you a network hop, a second deploy pipeline, a second failure domain, a versioning contract between two codebases, and the cognitive overhead of debugging across processes. For a small team, that tax is real and recurring.

You only earn that cost back when the two sides of the boundary need to change on different schedules. That is the entire point of service decomposition.

For an AI feature, that question reduces to one thing: does the model get updated independently of the application?

• If you call OpenAI or Anthropic and the prompt lives in your application code, model and app deploy together. No independent cadence. No boundary justified.
• If you fine-tune a model weekly on fresh production data, push new weights to a registry, and want to roll back inference without rolling back the app — independent cadence. Boundary justified.
• If you load a HuggingFace model at app startup and only update it when you upgrade the library version — together. No boundary.
• If you have a retraining pipeline that runs on a schedule, owned by a data scientist who does not touch the Rails repo — independent. Boundary justified.

This is the same logic that justifies extracting any service: different teams, different release cadences, different scaling needs. AI does not get a special exemption from that test. It also does not get a free pass to skip it.

The five axes that actually matter

1. Retraining cadence (the primary axis)

Covered above. If the model and app ship as one unit, the monolith wins. If the model has a life independent of the app — separate pipeline, separate registry, separate rollback — extract it.

For most first AI features at a Series A SaaS, the model is a hosted API call or a static off-the-shelf model. The cadence is coupled. Monolith.

2. Language and runtime fit

If you are calling a hosted API (OpenAI, Anthropic, Cohere, Bedrock), Ruby and Python both have clients. No runtime mismatch. Monolith.

If you need to run a model in-process — say, a sentence-transformers embedding, a sklearn classifier, a small ONNX model — and your monolith is Rails, the Python ecosystem gap is real. You can shell out, use a Ruby ONNX runtime, or run a sidecar. None of these is great. A separate Python service starts to make sense.

Django monoliths get a pass here. Python all the way down. Most ML libraries are first-class. The runtime argument for extraction is weaker than people think.

3. Resource profile (GPU, memory, cold start)

If inference needs a GPU and the rest of your app does not, mixing them on the same host is wasteful and operationally awkward. Auto-scaling a web tier on request volume and a GPU tier on inference queue depth are different problems.

But: most first AI features at this stage do not need a GPU. Hosted APIs do the heavy lifting. Small models run on CPU. If your model fits in 2 GB of RAM and responds in under 200ms on CPU, this axis does not apply.

If it does — if you are self-hosting Llama or running batch embeddings on a 7B model — extract. The resource profile difference alone justifies the boundary.

4. Latency budget and failure mode

An in-process call has microsecond overhead. An HTTP call adds 5–50ms in the same VPC, more across regions, and introduces timeout handling, retry logic, and a new failure mode (the model service is down but the app is up — what do you show the user?).

For synchronous, user-facing AI features (autocomplete, search ranking, in-product chat), every added millisecond matters and every new failure mode is one your support team will see. The monolith wins on both.

For async features (nightly summarization, batch scoring, background enrichment), latency and partial failure don't matter. Either architecture works. Pick by other axes.

5. Team shape

If the same engineer writes the app code and the inference code, the monolith removes friction. One repo, one PR, one deploy.

If you have a data scientist or ML engineer who does not touch the application repo, who works in notebooks, who owns the model — a service boundary lets them ship without coordinating every deploy. This is the team-shape version of the retraining-cadence argument, and they usually correlate.

At 4–8 engineers, you probably do not have this split yet. One person wears both hats. Monolith.

Scoring the two options honestly

What the monolith is bad at

• Dependency conflicts. Adding torch or transformers to a Rails or Django app bloats the image and can collide with existing pins. Build times go up. Container size goes up.
• Resource isolation. A runaway inference call can starve the web workers. You'll want per-request timeouts and probably a separate worker pool — which is a soft service boundary inside the monolith.
• Model artifact size. Shipping a 2 GB model in your application image is painful. You'll want to load from S3 or a registry at boot, which adds startup time and a new failure mode at deploy.
• It hides the real boundary. If the model genuinely does need independent cadence later, you'll have built coupling you have to unwind.

What the separate service is bad at

• Operational tax. Two pipelines, two on-call surfaces, two sets of dashboards, two sets of secrets. For a team that has never run distributed inference, this is a lot.
• Versioning contracts. The app and the service now have an API contract. Breaking changes need coordination. Schema drift is a real bug class you didn't have before.
• Local development. Engineers now need both services running to test end-to-end. Docker Compose helps. It doesn't make the problem go away.
• Premature abstraction. If you guess wrong about the service interface — and you will, because you haven't shipped the feature yet — the rewrite cost is higher than refactoring inside a monolith.
• Latency floor. You can never go below the network hop. For features where latency matters, this is a hard ceiling.

The rule

If you are in situation A, do X:

• First AI feature, hosted model API (OpenAI, Anthropic, Bedrock), Django or Rails monolith, no GPU, team of 4–8 → build it in the monolith. Put the inference code in a clearly-named module (app/ai/ or lib/inference/). Wrap external API calls in a thin client with timeouts and circuit breakers. Log every prompt and response. Ship.
• Self-hosted model, CPU inference, fits in RAM, no separate ML person, model updates only when you upgrade the library → build it in the monolith, but load the model in a dedicated worker pool (Sidekiq queue, Celery queue, or a separate process group). Treat it as a soft boundary you can harden later.

If you are in situation B, do Y:

• You have a real retraining pipeline that runs on its own schedule, owned by someone who doesn't touch the app repo → extract the service. The independent cadence justifies the cost.
• You need GPU inference and the rest of your app is CPU → extract. Mixing resource profiles is operationally worse than a service boundary.
• Your monolith is Rails and the model only runs well in Python, and you can't use a hosted API → extract. The runtime gap is real.
• The model is shared across multiple applications or product surfaces → extract. Code reuse across deploy units is a service.

The honest grey zone: if you're 60/40 toward extraction, stay in the monolith. The cost of being wrong inside a monolith is a refactor. The cost of being wrong with a service is two pipelines you maintain forever, plus a migration when the contract turns out wrong.

A practical pattern for staying in the monolith without regret

If you go with the monolith, structure the code so the future service boundary is easy to extract:

• One module, one entry point. All AI calls go through InferenceClient.predict(input) or equivalent. No scattered openai.chat.completions.create calls in 15 controllers.
• Treat the function signature as if it were an RPC contract. Inputs and outputs are plain dicts or dataclasses, not ActiveRecord objects. This makes a future HTTP boundary a sed command, not a rewrite.
• Wrap every external call in a timeout, a retry budget, and a circuit breaker. The hosted model will go down. Plan for it.
• Log inputs, outputs, model version, and latency for every inference. You need this for debugging, evals, and the eventual fine-tuning dataset.
• Version your prompts the same way you version migrations. A prompt change is a code change.

Do this and the monolith path stays cheap. If retraining cadence later diverges, extraction is mechanical.

How CodeNicely can help

We've shipped this exact decision more than once. With HealthPotli, we embedded an AI drug interaction checker into an e-pharmacy product that was already in production. The interaction model was a hosted call wrapped in business rules — same cadence as the app, no GPU, latency-sensitive at checkout. The right call was in-monolith with a clean inference module, not a separate service. The team shipped the feature without standing up a second pipeline, and the boundary stayed easy to extract if usage patterns later demanded it.

That is the pattern we recommend most often to Series A SaaS teams: ship the first feature where it costs least to be wrong, structure the code so extraction is cheap, and only pay the service tax when an independent retraining cadence actually shows up. If you'd like a second pair of eyes on your specific case, our AI studio does these architecture reviews as standalone engagements before any build work.

Frequently Asked Questions

Should I use microservices for AI if I'm already using microservices for everything else?

Yes — if the rest of your system is already microservices, the marginal cost of one more service is low and the consistency argument wins. The post is aimed at teams whose primary application is a monolith. If you already operate 10 services, the operational tax of a new one is mostly already paid.

What if I'm not sure whether the model will be retrained independently later?

Stay in the monolith and structure the inference code as if it were an RPC client — one entry point, plain-data inputs and outputs, no ORM objects crossing the boundary. If retraining cadence later diverges, extraction is a mechanical refactor, not a rewrite. The cost of guessing wrong toward the monolith is small. The cost of guessing wrong toward a service is recurring.

Does this advice change if I'm using LangChain or a vector database?

Not much. LangChain is a library, not a service — it runs wherever you put it. A managed vector DB (Pinecone, Weaviate Cloud) is already a separate service you call over the network, which doesn't change the decision about your own inference code. A self-hosted vector DB is its own infrastructure decision, separate from where the inference logic lives.

How do I handle model versioning if everything is in the monolith?

Pin the model identifier (API model name or local artifact hash) in config, not in code. Log the version with every inference. When you change models, treat it as a config change with a feature flag and a rollback plan. This gives you most of the version-control benefit of a separate model registry without the infrastructure.

How long does it take to extract an AI feature from a monolith into a service later?

It depends entirely on how the code was structured at the start — a clean inference module with dict in / dict out can be extracted quickly; scattered API calls across controllers are a much bigger job. For a real estimate against your codebase, contact CodeNicely for a personalized assessment.

The summary, one more time: split on retraining cadence. If the model ships with the app, keep it in the app. If the model has its own release life, give it its own service. Everything else is a tiebreaker.