Vector Databases for System Design Interviews

01Why Vector Databases Are Their Own Thing

Most candidates can name "Pinecone" and stop there. The interview goes deeper: what is an embedding, actually? Why is vector search approximate? What's the recall-vs-latency tradeoff in ANN algorithms, and how do you tune it? When does a dedicated vector store make sense versus pgvector inside Postgres? How does this all compose with keyword search and the rest of the system?

The depth here lives in three places. First, the conceptual foundation: embeddings are how text, images, and other content get translated into a geometric space where similarity becomes distance. Most candidates use this without understanding it. Second, approximation: exact nearest-neighbor search is too slow at scale, so production systems use approximate algorithms (HNSW, IVF) that trade recall for speed. The trade space matters. Third, architectural integration: vector search is rarely standalone; it sits inside a larger retrieval pipeline alongside keyword search, primary stores, and downstream LLM applications.

This page covers all three. Specific products are mentioned but they're not the point. The point is the underlying decisions, because the products will change every year and the decisions won't.

02What an Embedding Actually Is

Before talking about vector databases, you need to know what they're storing. A vector database stores embeddings: numerical representations of content (text, images, audio) in a high-dimensional space.

The conceptual leap

Take a sentence: "Tokyo restaurants are great." Run it through an embedding model. The model produces a list of numbers, often 768 or 1024 of them, that represent the sentence's meaning in a learned geometric space. A different sentence with similar meaning, like "Restaurants in Tokyo are excellent," produces a similar list of numbers — close to the first one in the geometric space. A sentence about something different, like "Quantum mechanics is complex," produces numbers far from the Tokyo sentences.

That's the conceptual leap: meaning gets encoded as geometric position. Similar things are close. Different things are far. Once you have that, "find similar content" becomes "find nearby vectors."

Where embeddings come from

Embedding models are neural networks trained to produce these geometric representations. Modern options:

• OpenAI embeddings. text-embedding-3-small and text-embedding-3-large. Hosted, paid, high quality. The default for most production systems.
• Cohere embeddings. Multilingual support, often outperforms on non-English content. Hosted, paid.
• Open-source models. sentence-transformers (BERT-based), BGE, E5, Nomic embed. Run them yourself, no per-call cost, full control over the pipeline. Quality has improved dramatically; the gap with paid options is small for most use cases.
• Domain-specific models. Code embeddings (CodeBERT), legal text embeddings, biomedical embeddings. Specialized models that outperform general-purpose ones on their domain.

The choice of embedding model matters more than the choice of vector database. A great vector database with a poor embedding model will retrieve poorly. A modest vector database with great embeddings will retrieve well. Most teams obsess over the database; they should obsess over the embeddings.

What dimensions and distance actually mean

An embedding has dimensions: how many numbers it contains. Common sizes: 384, 768, 1024, 1536, 3072. Higher dimensions can capture more nuance but cost more to store and search. Most production embeddings are 768 or 1536.

Distance between two embeddings measures dissimilarity. Three common functions:

• Cosine similarity. Measures angle between vectors, normalized for magnitude. The default for text embeddings. Range: -1 (opposite) to 1 (identical), with 0 being unrelated.
• Euclidean distance (L2). Straight-line distance. Sometimes used for image embeddings.
• Dot product. Cheaper to compute than cosine. Used when embeddings are pre-normalized; equivalent to cosine in that case.

For most applications, cosine similarity is the right default. The choice rarely matters dramatically; the embedding model matters far more.

The embedding model determines what's similar to what. The vector database stores and searches efficiently. Get the embedding right; the database is downstream of that decision.

03Vector Similarity and Approximate Nearest Neighbor

Once you have embeddings, the core query is "find the K vectors most similar to this query vector." This is k-nearest-neighbor search. The naive approach: compare the query vector to every vector in the database, return the K closest. This is exact but slow: O(N) per query, where N is the number of vectors. At a million vectors and 768 dimensions, every query touches 768 million floating-point comparisons. Too slow at scale.

Why approximation is necessary

Exact nearest-neighbor search has a real cost. For a corpus of 10 million vectors at 768 dimensions, every query requires roughly 10 million × 768 = 7.7 billion floating-point operations to find the exact top-K. That's hundreds of milliseconds even on fast hardware. At 100 million vectors, it's seconds per query. Production search needs to be tens of milliseconds.

The fix is approximate nearest neighbor (ANN) search: algorithms that return the top-K with high probability without scanning every vector. Modern ANN algorithms can search a 100M-vector corpus in a few milliseconds with 95%+ recall (the fraction of true nearest neighbors actually returned). The ~5% you miss are usually marginal results that wouldn't have changed user experience.

The interview move: when asked "how does vector search work at scale?", name the approximation explicitly. "Exact search is O(N) per query and too slow above a million or so vectors. Production systems use approximate nearest neighbor, typically HNSW, which trades a few percent of recall for 100x to 1000x speedup. The recall is tunable through algorithm parameters."

04ANN Algorithms: HNSW, IVF, and the Trade Space

Three algorithm families cover almost every production vector store. Each has a different position on the recall-vs-latency-vs-memory trade space. The right choice depends on which dimension matters most for your workload.

The recall-latency-memory trade space

Three dimensions you're trading off. Pick which two matter most:

• Recall. The fraction of true nearest neighbors actually returned. Higher is better. 95% recall means the system returns 95 of the actual top 100 nearest neighbors; the other 5 are missed.
• Latency. How long a single query takes. Lower is better. Production targets are usually 10-50ms for vector search.
• Memory. How much RAM the index occupies. Lower is better, but most ANN indexes have to fit in memory for fast queries.

The fundamental relationship: you can pick any two. Want high recall and low latency? Pay memory. Want low memory and high recall? Pay latency. Want low memory and low latency? Pay recall. Algorithms differ in how favorably they sit on this trade curve, but the curve itself is real.

Tuning HNSW

Two parameters dominate HNSW behavior:

• ef_construction (build-time): how thoroughly the graph is built. Higher values produce better graphs (better recall at query time) but slower indexing. Common range: 100-400.
• ef_search (query-time): how many candidates to consider during search. Higher values produce better recall but slower queries. Common range: 50-200. Tunable per-query, which is useful: high ef_search for important queries, lower for bulk operations.

The interview move: when asked about ANN tuning, name the trade space and at least one parameter. "We'd use HNSW with ef_search around 100 as the default; tune up for higher recall on important queries, tune down when latency dominates. The actual values are workload-specific and need empirical evaluation."

05The Retrieval Pipeline

Vector search rarely operates in isolation. It sits inside a larger pipeline: embed at ingestion time, store in a vector database, embed queries, search, return top-K. The diagram below shows the full flow.

Ingestion: chunking and embedding

Documents go through two steps before they reach the vector store:

• Chunking. Break documents into pieces small enough to embed effectively. Most embedding models have token limits (8K to 32K depending on the model); long documents need to be split. Common chunk sizes: 256-1024 tokens with overlap. The chunking strategy matters: too small and you lose context, too large and embeddings dilute.
• Embedding. Run each chunk through the embedding model. Store the resulting vector alongside metadata (document ID, chunk position, source URL, etc.) in the vector store.

Chunking is more art than science. The "right" chunk size depends on the content type, the embedding model, and the queries you expect. Most teams iterate: try a chunking strategy, evaluate retrieval quality, adjust. Recursive chunking (split on sentence boundaries, then paragraph boundaries, then with overlap) is a reasonable default.

Query: embed, search, optionally rerank

At query time, the user's query goes through the same embedding model used for ingestion. The vector store does ANN search and returns the top-K candidates. Many production systems then rerank: a more expensive model scores the candidates against the query for final ordering.

The reranking step is increasingly important. Initial retrieval (ANN) optimizes for recall: get the right candidates into the top-K. Reranking optimizes for precision: order the top-K so the best ones are first. Cross-encoder rerankers (like Cohere Rerank) score query-document pairs directly and dramatically improve ranking quality at modest cost.

The model-match invariant

The same embedding model must be used for both ingestion and query. Different models produce different geometric spaces; query embeddings from one model and document embeddings from another will be in incompatible spaces, and the search results will be garbage. Changing your embedding model means re-embedding the entire corpus. This is one of the highest-risk operations in a vector retrieval system; plan for it explicitly.

06pgvector vs Dedicated Vector Stores

The most common architecture decision in vector databases: do you use pgvector inside your existing Postgres instance, or do you stand up a dedicated vector store like Pinecone, Weaviate, or Qdrant? The honest 2026 answer is "pgvector first, dedicated when scale demands."

Why pgvector usually wins for new projects

Three reasons most products should start with pgvector:

• Operational simplicity. One database to operate, one set of backups, one set of credentials, one query interface. Adding a dedicated vector store doubles the operational surface for marginal gain at small to moderate scale.
• Joins with structured data. Most retrieval queries have filters: "find documents similar to this query, but only ones owned by this user, only published in 2026, only in English." Postgres handles this naturally through SQL. Many dedicated vector stores treat filtering as an afterthought, applied post-search, which can hurt result quality.
• Transactional consistency. Inserting a document and its embedding in one transaction is trivial in Postgres. Coordinating two systems (primary store + dedicated vector store) requires the same patterns from search-indexing — CDC pipelines, eventual consistency, sync drift to manage.

When dedicated stores actually win

Three scenarios where moving beyond pgvector makes sense:

• Scale. Beyond ~10-50M vectors, pgvector can struggle on a single Postgres instance. Dedicated stores are designed for sharding from day one.
• Specialized features. Multi-vector documents (storing multiple embeddings per document and querying them flexibly), sparse-dense hybrid scoring built-in, learned indexes, GPU acceleration. These come naturally to dedicated stores; pgvector doesn't have them.
• Latency requirements. Dedicated stores often have lower p99 latency at scale because they're not sharing resources with transactional workloads. If your retrieval is in a tight latency budget (sub-20ms), dedicated may be necessary.

07RAG: The Canonical Application Pattern

The most common reason to build a vector database in 2026 is RAG: Retrieval-Augmented Generation. The pattern is simple in outline: when a user asks a question, retrieve relevant documents from your corpus, include them as context in the LLM prompt, generate a grounded answer. The vector database is the retrieval engine.

Why RAG exists

LLMs have two failure modes that retrieval addresses:

• Knowledge cutoffs. The LLM only knows what was in its training data, which has a cutoff date. Anything newer is invisible. Retrieval brings in current information.
• Hallucination. LLMs confidently make up facts that sound plausible. Grounding the response in retrieved documents (and asking the LLM to cite them) reduces hallucination dramatically.

RAG isn't perfect. It can fail when retrieval misses the relevant document, when the LLM ignores the retrieved context, when the retrieved context is contradictory or low-quality. But it's much better than asking an LLM directly about content it doesn't know.

The pieces of a production RAG system

The basic pattern (retrieve, prompt, generate) is the start. Production RAG systems add several layers:

• Hybrid retrieval. Vector search alone misses exact keyword matches. Production systems combine vector search (semantic) with keyword search (BM25) using reciprocal rank fusion. Search and indexing covers the hybrid pattern.
• Reranking. Initial retrieval returns top-K candidates; a more expensive cross-encoder reranks them by relevance. The compute cost is justified because reranking quality directly affects answer quality.
• Query rewriting. The user's raw query may not match document phrasing well. Some systems rewrite the query (often via the LLM itself) before retrieval. Multi-query retrieval generates several variations and combines results.
• Citation and grounding. The LLM is prompted to cite retrieved documents. Post-processing verifies the citations are real. Hallucinated citations are a known failure mode worth detecting.

The depth probe

"How would you build a customer support chatbot over our knowledge base?" is a common variant. The strong response covers the full pipeline: embed the knowledge base into pgvector, hybrid retrieval combining BM25 with vector search, rerank the top-K, prompt the LLM with the top results as context, return the answer with citations. Mentioning observability (per-query latency, retrieval recall, LLM token cost) and the model-match invariant is what makes the answer staff-bar.

08Failure Modes

09How Vector Databases Interact With Other Concepts

• Vector databases × Search and indexing. The hybrid keyword + vector pattern from search and indexing is the dominant 2026 architecture. Keyword search and vector search compose; this page covers the vector half, search-indexing covers how they combine.
• Vector databases × Database selection. Vector store is its own database category in database selection. The choice between pgvector and dedicated stores is a database selection question: scale, features, operational fit. Postgres-as-default applies here too.
• Vector databases × Sharding. Vector indexes shard the same way primary stores do, with the same hot-key risks. Sharding covers the analogous tradeoffs; the partition key for vector data is usually tenant or document type.
• Vector databases × Caching. Hot queries (the same question asked many times) can be cached. Embedding the same query twice produces the same vector; caching the embedding step alone removes most of the per-call LLM cost. Caching covers placement.
• Vector databases × Observability. Vector retrieval has unique metrics: recall (often hard to measure on production traffic), embedding latency, ANN search latency, reranker latency, total token cost for LLM calls. Observability covers the broader pattern; vector workloads need specific instrumentation.

For more cross-concept interactions, see the concepts library hub.

10Practice Scenarios

Three scenarios. Read the setup. Decide your approach before opening the reveal.

11Vector Databases FAQ