Evaluating embedding models: a practical guide

A public leaderboard is a reasonable place to start choosing an embedding model, and a bad place to stop. Benchmarks like MTEB average performance across dozens of tasks and domains — retrieval, classification, clustering, across news, code, and general web text. A model that ranks near the top overall can still underperform quietly on the one thing that actually matters: retrieval quality on your own corpus, in your own domain's vocabulary. A model trained mostly on general web text doesn't necessarily represent financial filings, medical notes, or legal contracts well, and a strong average score gives no visibility into that gap.

What actually varies between models

Dimensionality is the first practical difference — anywhere from 384 to 3072 dimensions depending on the model — and it isn't free: every dimension is storage and index memory multiplied by however many chunks a corpus has, and it affects search speed at scale. Cost per token varies by an order of magnitude or more between a provider's smaller and larger embedding models, and self-hosting an open-source model (BGE, E5, and similar) trades a per-token bill for GPU inference cost and the operational work of running it — not free, just a different kind of cost. Context window limits differ too: some models truncate well before a full chunk, others handle several thousand tokens without issue, which matters directly for whatever chunking strategy sits upstream of it. And only open-source models can be fine-tuned on your own labeled data — a proprietary API model is fixed at whatever the provider shipped, for better or worse.

Building an evaluation harness instead of trusting a leaderboard

The only evaluation that actually answers "which model works for us" runs candidate models against the same corpus, the same chunking, and the same query set — changing only the embedding model, so any difference in results is attributable to that one variable. That needs a labeled set: pairs of a real query and the document (or chunk) that's actually the right answer to it — the same "ground truth has to come from somewhere real" requirement that any evaluation runs into, curated by someone who knows the domain, not generated by having a model grade itself. With that in hand, the metrics that matter are about whether the right document got found at all, not about the quality of a generated answer: recall@k (is the right document anywhere in the top k results), and rank-aware metrics like MRR or NDCG, which additionally penalize a right answer buried at position 40 versus surfaced at position 2. This sits upstream of the generation-quality metrics — faithfulness, answer relevance — that matter once retrieval has already succeeded; a model can be excellent at generation and still fail if retrieval never found the right source to begin with.

Cost isn't only the per-token price

Ingestion cost is the number people budget for and the one that keeps recurring: embedding an entire corpus isn't a one-time expense if chunking changes, a bug gets fixed in extraction, or the corpus keeps growing — every reprocessing event re-embeds everything, not just what changed, because vectors from different embedding runs (even the same model, in some cases) aren't guaranteed comparable. Query-time embedding cost is smaller per request but scales with traffic, and self-hosting shifts the entire cost structure from a predictable per-token bill to infrastructure that needs to be sized, monitored, and kept available — a real tradeoff, not a free upgrade just because it removes a line item from a vendor invoice.

When switching is actually worth it

Migrating embedding models is never a config change — it's re-embedding the full corpus, because vectors from two different models don't live in a comparable space and can't be mixed in the same index. That makes the decision to switch a real cost, which means it's worth making deliberately: run the same evaluation harness against the candidate model, on the same corpus, and switch when there's a measured, meaningful gap in recall or ranking quality on real queries — not because a new model topped a public leaderboard this month. The model that wins on a general-purpose benchmark and the model that wins on a specific corpus are frequently not the same model, and the only way to know which situation you're in is to measure it directly.