What Is a Vector Database? Stop Searching Your Data Like It's 2005

For: A product lead at a 50-person B2B SaaS company who just watched their engineering team spend three weeks tuning Elasticsearch relevance scores — and the search results still return garbage when users phrase queries differently than the docs were written

A vector database stores data as numerical representations of meaning (embeddings) instead of as exact tokens, which lets you retrieve results by semantic similarity rather than keyword match. If your users search for "cancel my subscription" and your docs say "terminate account," a keyword index returns nothing useful and a vector search returns the right doc. That's the whole pitch. Everything below is about when it's worth the added infrastructure and when you're better off tuning what you have.

If your team just spent three weeks tuning Elasticsearch relevance scores and users still can't find what they need, you're hitting a structural ceiling — not a configuration problem. Boosting rules, synonym files, and BM25 tweaks can only push a token-matching system so far.

The problem: keyword search doesn't understand meaning

Traditional search — Elasticsearch, Solr, Postgres full-text, Algolia — is fundamentally a lookup on tokens. You index words. You search for words. Relevance is scored on frequency, position, and inverse document frequency. It's fast, deterministic, and it has powered search for two decades.

It also breaks the moment a user's vocabulary diverges from your documents' vocabulary. A few common failure modes:

• Synonyms: "laptop" vs "notebook," "refund" vs "reimbursement," "login issue" vs "can't sign in."
• Paraphrase: "how do I stop getting billed" vs "how to cancel subscription."
• Cross-language: A French customer typing "annuler mon abonnement" against English docs.
• Intent mismatch: "my dashboard is empty" is really a question about data sync latency.

You can patch some of these with synonym dictionaries and query expansion. But you are hand-authoring meaning, one rule at a time, and the maintenance cost compounds with every new product feature. This is the wall your engineering team just hit.

The concept: encode meaning as coordinates

Here is the analogy that makes it click. Imagine a map of every concept your product handles. Words and phrases that mean similar things live near each other. "Cancel subscription," "terminate account," and "stop billing" are clustered in one neighborhood. "Reset password" and "forgot login" are in a different neighborhood across town.

A vector embedding is the coordinates of a piece of text on that map. Except the map has 384, 768, or 1,536 dimensions instead of two. An embedding model (OpenAI's text-embedding-3-small, Cohere's embed-v3, open-source models like bge-large or e5) reads text and outputs a fixed-length array of floats — the coordinates.

A vector database is the thing that stores those coordinate arrays and answers the question: "given this new point, what are the nearest N points I've already stored?" That's it. Semantic search is nothing more than nearest-neighbor lookup in high-dimensional space.

The trick is doing this quickly across millions of vectors. Naive nearest-neighbor search is O(n). Vector databases use approximate nearest neighbor (ANN) algorithms — HNSW, IVF, ScaNN — that trade a tiny amount of recall for orders-of-magnitude speedup.

A minimal worked example

Say you run a B2B SaaS help center with 2,000 articles. Here's the flow with a vector embeddings database:

1. Index once: For each article, call an embedding model. You get back a 1,536-dim vector per article. Store the vector + article ID in a vector database (Pinecone, Weaviate, Qdrant, pgvector, Milvus).
2. At query time: User types "why is my invoice wrong." Embed that query with the same model. You get a 1,536-dim vector.
3. Search: Ask the vector database for the 10 nearest article vectors to the query vector. It returns article IDs ranked by cosine similarity.
4. Return results: Fetch the article contents from your primary database (Postgres, Mongo — whatever) and render.

Notice what didn't happen: no keyword matching. The article titled "Understanding your billing statement discrepancies" scores highly because its embedding lives near the query's embedding on the meaning-map, even though it shares zero important words with "why is my invoice wrong."

Where vector search fits alongside what you already have

Vector databases don't replace your Postgres or your Elasticsearch. They sit next to them. Most production systems use hybrid search: run both a keyword query and a vector query, then merge the results with a reranker or a weighted score. Keyword search still wins for exact matches — SKU lookups, error codes, proper nouns — where semantic similarity is noise. Vector search wins on natural language.

This is also the foundation of Retrieval-Augmented Generation (RAG). When you see an AI assistant that "knows your docs," what's happening under the hood is: embed the user question, vector-search your docs, stuff the top-k results into an LLM prompt as context. The vector database is the retrieval layer. Without it, the LLM either hallucinates or is limited to whatever fits in its context window.

Gotchas nobody mentions in the vendor pitch

• Embeddings are model-locked. Vectors from OpenAI's model are not comparable to vectors from Cohere's model. If you switch embedding providers, you re-embed your entire corpus. Budget for this.
• Chunking matters more than the database. A 40-page PDF embedded as one vector is useless — the meaning is averaged out. You need to split documents into semantically coherent chunks (200–800 tokens is a common range) with some overlap. Bad chunking beats bad vector DB choice as the top cause of poor retrieval.
• Semantic similarity is not relevance. Two documents can be "about the same topic" but one is outdated, wrong, or for a different customer tier. You still need metadata filters (tenant ID, published date, product version) and often a reranking step.
• Cost scales with dimensionality and volume. 1,536-dim vectors × 10M documents is a lot of RAM. Look at quantization, dimension reduction, and tiered storage before you're surprised by the bill.
• Freshness is your job. When a doc is updated, you re-embed and upsert. If your product changes weekly, wire this into your CMS or deploy pipeline from day one.
• Evaluation is hard. Unlike keyword search where you can eyeball results, semantic quality requires a labeled eval set (query → expected top result) and a recall@k metric. Without it, you cannot tell if changing the embedding model helped or hurt.

When to use a vector database — and when not to

Use one when:

• Users search in natural language and your docs are written in product-speak.
• You're building a RAG-powered assistant, chatbot, or copilot on your own content.
• You have multilingual users and single-language content (or vice versa).
• You're doing semantic deduplication, recommendation, or clustering on unstructured text.
• Your relevance-tuning backlog on Elasticsearch has become a full-time job.

Don't bother when:

• Your corpus is small (a few hundred documents) and structured — a good keyword index with synonyms will match a vector setup and cost less to operate.
• Search is primarily on structured fields: SKUs, order IDs, user emails, categorical filters.
• You haven't measured your current search failure rate. "It feels bad" is not a spec. Log the queries with zero clicks first.
• You need strict, explainable ranking (regulated industries, compliance search). Vector similarity is a black box compared to BM25.

The honest answer for most B2B SaaS teams is hybrid: keep your existing search, add a vector layer for the queries where it obviously fails, and merge. Rip-and-replace is rarely the right call.

Which vector database

Quick orientation, no rankings:

• pgvector — a Postgres extension. If you're already on Postgres and have under ~10M vectors, start here. One less system to run.
• Qdrant, Weaviate, Milvus — open-source, self-hostable, feature-rich (filters, hybrid search built-in).
• Pinecone — fully managed. Fastest path to production if you don't want to run infrastructure.
• Elasticsearch / OpenSearch — both now support dense vector fields. If you're already invested, the incremental step is small.

The database choice matters less than the embedding model, the chunking strategy, and the eval harness. Teams that get semantic search wrong almost always got those three things wrong first.

If you're weighing whether a vector search layer is worth adding to your product — or whether your problem is really an embeddings-and-chunking problem in disguise — that's the kind of scoping work our AI studio team does before writing a line of code.

Frequently Asked Questions

What's the difference between semantic search and keyword search?

Keyword search matches exact tokens (or their stems) between a query and indexed documents, scored by frequency. Semantic search compares the meaning of the query and documents, represented as vector embeddings, using distance in high-dimensional space. Keyword search wins on exact identifiers and structured lookups; semantic search wins on natural language, paraphrase, and cross-language queries.

Do I need a vector database if I'm already using Elasticsearch?

Not necessarily. Recent versions of Elasticsearch and OpenSearch support dense vector fields and k-NN search natively, so you can add semantic search without introducing a new system. A dedicated vector database (Pinecone, Qdrant, Weaviate) is worth it when you need higher-scale ANN performance, advanced filtering on vectors, or you're building fresh without Elasticsearch investment.

Can a vector database replace my primary database?

No, and it's not designed to. A vector database is a specialized index for similarity search. You still need your Postgres, MongoDB, or equivalent as the source of truth for structured data, transactions, and relationships. Vectors reference back to primary records by ID.

How much data do I need before a vector database is worth it?

Less about volume, more about query patterns. If users are typing natural-language questions and your keyword search returns zero or wrong results more than ~10–15% of the time, semantic search is worth testing — even at a few thousand documents. Below a few hundred documents, invest in good synonyms and metadata first.

How do I estimate the effort to add semantic search to our SaaS product?

It depends on corpus size, how your content is structured, whether you need multilingual support, and whether you're building RAG on top or just search. For a scoped estimate on your specific stack and data, contact CodeNicely for a personalized assessment.