Vector databases

Once every chunk is a vector, you need to store them and find the nearest ones to a query vector. The industry answer is 'a vector database' — but the honest engineering answer starts smaller, and knowing the ladder saves you from the most common RAG mistake: buying infrastructure before you have a retrieval-quality problem.

The storage ladder

1. A numpy array in memory. Exact nearest-neighbor search over 100,000 vectors takes ~tens of milliseconds on a laptop. HarborDocs (a few hundred chunks) lives here all course, and plenty of production systems with modest corpora rightly never leave.
2. Your existing database, extended — e.g. pgvector in Postgres. Vectors sit next to the rows they describe: one system, real filters, real transactions. The default answer for most companies. (You met this philosophy in our products: use the warehouse you already trust.)
3. A dedicated vector store — local ones like Chroma for development; managed services when you need scale. Earned when you have millions of vectors, high query volume, or heavy metadata-filtered search.

This course uses only a numpy array end to end; pgvector/Chroma are context for later scale, not something you install now.

The trade every real store makes: ANN

Exact search compares the query to every vector — perfect recall, linear cost. At millions of vectors, stores switch to approximate nearest neighbor (ANN) indexes (HNSW is the one you'll see everywhere): graph structures that find almost always the true nearest neighbors in a fraction of the time. The word to respect is approximate — an ANN index can miss the right chunk that exact search would have found. When you evaluate retrieval in Module 3, you're evaluating the index too.

What actually matters when choosing

• Metadata filtering — 'nearest chunks WHERE product_line = kitchen AND version = current'. You'll design these fields in Module 2; make sure your store filters efficiently, not as an afterthought.
• The update story — documents change. Can you upsert and delete by document id without rebuilding the world? (Module 7 pain, chosen now.)
• Operational fit — a vector store is a database: backups, access control, monitoring. The fewer new systems, the better your on-call life. Hence the pgvector default.