Gen AI

Vector Databases

A vector database wraps an approximate-nearest-neighbor index with the operational layers a raw index lacks — filtering, multi-tenancy, and persistence — turning a search structure into a real system.

A vector database wraps an approximate nearest-neighbour index with the operational layers a raw index lacks — metadata filtering, multi-tenancy, and persistence — so it functions as a real system of record, not just a search structure

Why Does This Exist?

An approximate nearest neighbor algorithm, on its own, is a data structure and a search procedure — build an index over a fixed set of vectors, then find the closest ones to a query. That's necessary but nowhere near sufficient for a real production system. Real applications need to insert new documents continuously, delete outdated ones, restrict a search to only the current user's own data, filter results by a date range or a category tag, and survive a server restart without rebuilding the entire index from scratch. None of that is the job of the underlying nearest-neighbor algorithm itself.

A vector database is the system that wraps a nearest-neighbor index with everything else a production application actually needs around it. It's the difference between having a fast search algorithm and having something you can actually build a real, operable product on top of.

Think of It Like This

An engine versus a car

An approximate-nearest-neighbor algorithm is an engine: genuinely the part that makes movement possible, and the component most people are curious about when comparing options. But nobody drives a bare engine to work. A car wraps that engine with a fuel system, brakes, a steering column, a body that survives a crash, and a dashboard that tells the driver what's actually happening — none of which the engine provides on its own, and all of which are what makes the engine usable for an actual trip.

A vector database is the car. The ANN algorithm inside it — whichever one a specific product chose — is genuinely important, but swapping engines rarely changes what the rest of the car needs to do.

How It Actually Works

The layer beneath: an ANN index

At the base of every vector database sits some approximate-nearest-neighbor algorithm — commonly a graph-based method or a partition-based one, each with its own recall/latency/memory tradeoff. This layer answers exactly one question fast: given a query vector, which stored vectors are closest to it. Everything above it in the diagram is what makes that one capability usable in a real system.

Metadata filtering: searching only the relevant subset

Real queries are rarely "find the nearest vectors, full stop" — they're "find the nearest vectors among documents from the last 30 days" or "among documents this specific user is permitted to see." Metadata filtering attaches structured fields to each stored vector and applies filters before or alongside the similarity search. The ordering matters: filtering before the nearest-neighbor search (pre-filtering) guarantees every candidate the search considers is valid, but can hurt recall if the filter is highly restrictive and the search structure wasn't built with that in mind; filtering after (post-filtering) is simpler to implement but can return too few results if most of the nearest neighbors get filtered out afterward. Which approach a given database uses is a real design decision with real consequences for recall.

Multi-tenancy: keeping one customer's data invisible to another's

For any product serving multiple customers or users from one shared system, multi-tenancy ensures one tenant's vectors are never mixed into another tenant's search results — a strict requirement, not a nice-to-have, for anything handling separate customers' data. This is implemented as a specific isolation mechanism (separate indexes per tenant, or a mandatory tenant-id filter baked into every query) rather than left as an optional filter a developer might forget to apply.

Persistence: surviving a restart

An in-memory-only index loses everything on a crash or restart, which is unacceptable for any real system of record. Persistence means the vectors, their metadata, and the index structure itself survive process restarts — typically through some combination of writing to disk, replication across machines, and a durable write path that doesn't lose data mid-write. This is exactly the operational layer a bare ANN algorithm, evaluated purely on search speed, has no reason to provide on its own.

Show Me the Code

A minimal illustration of pre- versus post-filtering, showing exactly why the ordering changes the result count.

1import numpy as np2
3
4def post_filter(candidates: list[tuple[int, float, str]], allowed_tenant: str, k: int) -> list[tuple[int, float, str]]:5    """Search first, THEN filter -- may return fewer than k if most neighbors are filtered out."""6    ranked = sorted(candidates, key=lambda c: c[1], reverse=True)[:k]7    return [c for c in ranked if c[2] == allowed_tenant]8
9
10def pre_filter(candidates: list[tuple[int, float, str]], allowed_tenant: str, k: int) -> list[tuple[int, float, str]]:11    """Filter first, THEN search -- always returns k results if enough exist for this tenant."""12    eligible = [c for c in candidates if c[2] == allowed_tenant]13    return sorted(eligible, key=lambda c: c[1], reverse=True)[:k]14
15
16# (id, similarity score, tenant) -- only 1 of the top 5 nearest neighbors belongs to "tenant_a"17candidates = [(1, 0.95, "tenant_b"), (2, 0.93, "tenant_b"), (3, 0.91, "tenant_a"),18              (4, 0.89, "tenant_b"), (5, 0.87, "tenant_b")]19
20print(len(post_filter(candidates, "tenant_a", k=5)))   # -> 1 -- most of top-5 got filtered away21print(len(pre_filter(candidates, "tenant_a", k=5)))     # -> 1 too here, but only because there's just 1 eligible

Both return one result in this small example, but at real scale the gap is stark: post-filtering over a large candidate set can silently return far fewer than the requested k results whenever a filter is restrictive, while pre-filtering guarantees k results whenever k eligible candidates actually exist.

Watch Out For

Treating a bare ANN library as production-ready

An open-source nearest-neighbor library is often fast and genuinely well-built for the one job it does — searching a fixed, in-memory set of vectors. Deploying it directly as a production backend, without adding metadata filtering, tenant isolation, or persistence on top, works fine in a demo and then fails in exactly the ways this page describes the moment real usage begins: data loss on restart, one customer's results leaking into another's, or no way to scope a query to a relevant subset.

Assuming pre-filtering is always the safer default

Pre-filtering guarantees correctness in principle, but a highly restrictive filter combined with a graph-based ANN index can force the search to traverse far more of the graph than expected to find enough matches, sometimes degrading latency more than post-filtering would have. Neither approach is unconditionally better — the right choice depends on filter selectivity and the specific index structure underneath, and it's worth measuring both rather than assuming pre-filtering is automatically the safe choice.

The Quick Version

A vector database wraps a nearest-neighbor search algorithm with the operational layers a bare index lacks.
Metadata filtering restricts a search to a relevant subset, and whether it happens before or after the nearest-neighbor search has real recall consequences.
Multi-tenancy strictly isolates one tenant's vectors from another's, implemented as a structural guarantee rather than an optional filter.
Persistence ensures vectors and the index itself survive a restart, which a bare in-memory ANN algorithm doesn't provide by default.
The underlying ANN algorithm matters, but swapping it rarely changes what the layers built around it actually need to do.