Vector Search Is Not Semantic Search (And the Difference Will Break You)

For: A product or engineering lead at a mid-stage B2B SaaS company who added vector search to their product's search feature after reading about RAG, shipped it, and is now fielding complaints that it confidently returns wrong results for exact-match queries their old keyword search handled perfectly

Vector search is not semantic search, and treating them as interchangeable is why your product search now confidently returns the wrong SKU. Vector search finds the nearest embedding in high-dimensional space. Semantic search is the outcome you want — matching meaning. Vectors are one way to get there, but they optimise for neighbourhood, not correctness. For any query where precision matters more than recall — product codes, IDs, proper nouns, exact model numbers — a nearest-neighbour lookup will happily hand back a plausible-looking wrong answer instead of an honest empty result. Keyword search fails loudly. Vector search fails quietly. That difference will break you.

The problem vector search actually solves

Traditional keyword search (BM25, TF-IDF, Postgres full-text) matches tokens. Search for "laptop bag" and you get documents containing the words "laptop" and "bag". Search for "notebook sleeve" and you get nothing, even though a human would call that the same thing.

Embeddings fix that. A model converts text into a vector — a list of numbers, typically 384 to 3072 dimensions — where semantically similar text ends up geometrically close. "Laptop bag" and "notebook sleeve" land near each other. You store these vectors in a database (pgvector, Pinecone, Weaviate, Qdrant, Milvus), and at query time you embed the user's query and return the K nearest vectors by cosine similarity.

That's it. That's the whole trick. It's genuinely useful for fuzzy, intent-driven queries where the user's words don't match your document's words.

The intuition: a library organised by vibe

Imagine a library where books aren't shelved by title or author but by feel. Books about grief sit next to books about autumn. Books about competitive chess sit near books about military strategy. Ask for "something melancholy about endings" and the librarian walks you to a good shelf.

Now ask that same librarian for "ISBN 978-0-13-468599-1". She walks you to a shelf of books with similar-looking ISBNs. One of them is probably the right one. Probably.

That's vector search. It is structurally incapable of saying "I don't have that." The nearest neighbour always exists. Whether it's correct is a separate question the geometry can't answer.

A minimal worked example

Say you run a B2B parts catalogue. A user searches for BRK-4471-A. The correct part exists. So does BRK-4471-B (a different revision) and BRK-4741-A (a completely different part, transposed digits).

Here's what happens under the hood with a typical embedding model:

1. Query BRK-4471-A gets embedded. Most embedding models tokenise product codes into sub-word fragments that carry almost no semantic signal.
2. The three candidate codes embed into vectors that are extremely close to each other — the model sees them as "short alphanumeric strings starting with BRK."
3. Cosine similarity between the query and all three is roughly 0.94, 0.93, 0.93.
4. Your ranker returns them in an order that's essentially noise.

The user clicks the top result, orders the wrong revision, and files a support ticket. Your old WHERE sku = 'BRK-4471-A' would have returned exactly one row or zero. The vector version returned three, ranked confidently.

Gotchas that will bite you

1. Product codes, SKUs, and IDs are anti-semantic

Embeddings compress meaning. Identifiers have no meaning to compress — they're arbitrary strings whose value is exact-match equality. Route these to a keyword index or a direct database lookup. Never send them through a vector.

2. Numeric queries are worse than you think

"Motors under 500 watts" will not filter on 500 watts. The embedding of "500" and "5000" is close. Numeric ranges belong in structured filters, not in the vector.

3. Proper nouns hallucinate

Searching for a person, company, or product name that the embedding model has never seen produces near-random neighbourhoods. This is where you get the classic "user searched for ‘Acme Robotics’, got ‘ACME Logistics’ ranked first, and everyone downstream assumed it was right."

4. Similarity thresholds are not portable

A cosine score of 0.82 might mean "great match" in one corpus and "garbage" in another. Thresholds have to be tuned per index, per model, per query type. Change the embedding model and every threshold in your system is now wrong.

5. Chunking silently destroys context

If you're doing RAG, the chunk boundary decides what gets retrieved. Split a paragraph mid-sentence and the vector represents half a thought. Most "the retrieval is bad" problems are actually "the chunking is bad" problems.

6. Confidently wrong beats honestly empty — but only for the vendor, not the user

This is the one that catches product teams. Users tolerate "no results" because it's an honest signal. They do not tolerate a top result that looks right and isn't. Vector search has no null.

The fix: hybrid search, and route by query type

The teams that get this right don't pick a side. They run both and merge.

1. Classify the query at ingress. Does it contain a token that looks like a SKU, ID, email, part number, or exact phrase in quotes? Route to keyword/exact-match first.
2. Run BM25 and vector in parallel. Elasticsearch, OpenSearch, Vespa, and pgvector-plus-tsvector all support this. Merge results with Reciprocal Rank Fusion — a boring, well-understood algorithm that just works.
3. Apply structured filters before similarity. Category, price, in-stock, region. Do these in SQL, not in the embedding.
4. Set a floor on similarity. If nothing crosses your threshold, return empty. Give the user the honest null. This is the single highest-leverage change most teams skip.
5. Rerank the top 50 with a cross-encoder (e.g. Cohere Rerank, bge-reranker). Cheaper than you'd guess, and it fixes most of the "three near-identical scores" problem.

When to use vector search — and when not to

Use it for: free-text search over descriptions, docs, tickets, or knowledge bases; "more like this" recommendations; queries where the user's vocabulary won't match yours; the retrieval step in a RAG pipeline over prose.

Do not use it as the primary index for: product catalogues keyed on SKU, any lookup by ID or exact identifier, legal or compliance search where a missed exact match is a bug, structured data with clean fields, or anything a competent WHERE clause already handles.

The mistake isn't adopting vector search. It's replacing keyword search with it. Keyword search finds truth. Vector search finds meaning. A serious product needs both, wired up so the query gets to the right one. If you're rebuilding a search stack or an AI-powered feature that has to work across messy real-world queries, this hybrid architecture is the boring, correct default.

A short checklist before you ship

• Do you have a test set of 50+ real user queries, including exact-match cases, that you regression-test on every embedding or index change?
• Does your system return an empty result set when nothing crosses a similarity threshold?
• Are IDs, SKUs, and numeric filters handled outside the vector path?
• If you swap the embedding model, do you know which thresholds and prompts break?
• Are you logging query, retrieved IDs, and click-through so you can measure "confidently wrong" in production?

If you answered no to two or more, your users are finding bugs faster than you are.

Frequently Asked Questions

What's the difference between semantic search and vector search?

Semantic search is the goal: matching by meaning rather than exact tokens. Vector search is one implementation technique that uses embeddings and nearest-neighbour lookup to approximate that goal. Vector search can deliver semantic search, but it can also fail semantically — for example on identifiers, numbers, or unfamiliar proper nouns — where the geometry doesn't reflect meaning.

Do I need a dedicated vector database or is pgvector enough?

For most B2B SaaS workloads under a few tens of millions of vectors, pgvector on Postgres is enough and keeps your operational surface small. Dedicated vector databases (Pinecone, Weaviate, Qdrant, Milvus) earn their keep at higher scale, with heavier filtering, or when you need advanced index types like HNSW with fine-grained tuning. Start with what you already run.

Why does my vector search return confident but wrong results for product codes?

Embedding models tokenise alphanumeric identifiers into fragments that carry almost no semantic signal, so different codes end up as near-neighbours in vector space. The nearest-neighbour algorithm always returns something, so it returns a plausible-looking wrong code instead of nothing. Route identifier-like queries to exact-match keyword search, not the vector index.

Is hybrid search worth the added complexity?

For any product where users mix exact-match queries (IDs, names, part numbers) with free-text queries, yes. Reciprocal Rank Fusion over BM25 and vector results is a small amount of code and closes the biggest quality gap in most production search stacks. If your queries are purely conversational — like a chatbot over documentation — pure vector search may be enough.

How do I know if my current vector search implementation is broken?

Build a labelled test set from real user queries, especially ones your old keyword search handled well, and measure precision at rank 1. If exact-match queries score below 90% precision, or if your system never returns empty results, you have the failure mode described here. For a deeper audit of an existing search or RAG stack, contact CodeNicely for a personalized assessment.