Choosing an Embedding Model Without a PhD: What Retrieval Quality Really Depends On

An embedding model decides what counts as "similar" for your product, so retrieval quality depends on four concrete things: domain fit (understanding your jargon), language coverage (the languages your users write in), dimensionality (storage and search cost), and cost (API pricing or hosting load). Leaderboard rank captures none of them.

Quick answer: Don't pick an embedding model by its rank on a public leaderboard. Test 2-3 candidates against a small, labeled set of real queries from your own corpus, score precision@k and recall@k, and let domain fit and cost — not benchmark prestige — decide.

What Retrieval Quality Actually Depends On

Retrieval quality depends on whether the model's notion of "similar" matches your users' notion of "relevant" — and that match is governed by four concrete, product-level variables, not model prestige. Get domain fit, language coverage, dimensionality, and cost wrong, and no amount of prompt tuning downstream will fix what's already missing from the retrieved set.

An embedding model turns text — a support ticket, a product spec, a customer message — into a vector: a list of numbers positioned in space so that similar meaning sits close together. Retrieval is three mechanical steps built on top of that:

1. Embed every document in the corpus once, ahead of time.
2. Embed the user's query the same way, at request time.
3. Fetch the nearest vectors by cosine similarity and hand them downstream.

Everything after that, including any reranking step you add later, works on whatever this first pass hands it. That first pass is a context engineering decision, not a prompt engineering one — it happens before a single instruction reaches the model. If that distinction feels new, the breakdown of context vs. prompt engineering is worth reading first, since most "why is my AI feature giving weird answers" debugging sessions start in the wrong layer.

The Trap: Why Leaderboard Rank Doesn't Predict Your Retrieval Quality

The trap is treating the MTEB (Massive Text Embedding Benchmark) leaderboard as a shopping list. MTEB, introduced by Muennighoff and colleagues in 2022 and hosted on Hugging Face, standardized comparison across retrieval, clustering, and classification in dozens of languages — genuinely useful for a shortlist. But it's an average across public, general-domain datasets, and a model's average rank tells you almost nothing about how it performs on your specific documents and your users' actual queries.

This is exactly the failure mode the BEIR benchmark (Thakur et al., 2021) was built to expose. BEIR evaluates models zero-shot across more than a dozen genuinely different domains — biomedical questions, financial QA, argument retrieval, code search — and its consistent finding is that in-domain accuracy is a poor predictor of accuracy on an unfamiliar domain. A model topping a general leaderboard can rank well below a smaller, less-hyped one once the domain shifts to legal contracts or internal API documentation.

Three reasons the leaderboard number misleads technical PMs specifically:

1. Public benchmarks skew toward web text and Wikipedia-adjacent prose. Your corpus of support macros, internal runbooks, or clinical notes looks nothing like that.
2. "State of the art" changes monthly, and today's leaderboard topper is often a model tuned to squeeze out the last point of benchmark score, not to generalize.
3. Aggregate scores bury per-domain variance. A model can be excellent at semantic similarity and mediocre at retrieval — two different MTEB task categories that get averaged together.

None of this means benchmarks are useless — they're a reasonable way to shortlist three or four candidates worth actually testing. The mistake is stopping at the shortlist and calling it a decision.

Domain Fit and Language Coverage: The Levers Generic Comparisons Miss

Domain fit and language coverage are the two variables a spec sheet won't show you, because they depend entirely on what your users type and which language they type it in — not on anything the model card advertises. Both require you to look at your own data before you look at any comparison chart.

Domain Fit: Does It Know Your Vocabulary?

A general-purpose embedding model treats "churn" as a word about customers leaving — not, for a dairy client, a word about milk fat separating, and not, for a fintech client, a specialized reference buried in a compliance filing. Domain fit is whether the model's learned associations match the ones your documents actually rely on.

Symptoms of a domain-fit gap show up as specific, recognizable failures:

• Acronyms and internal shorthand retrieve unrelated results, because the model has never seen your glossary.
• Near-duplicate documents (two policy versions, two product SKUs) get ranked as equally relevant when only one is current.
• Numerically or structurally similar text (two ticket templates) outranks the ticket that's actually about the user's problem.

You don't need a custom-trained model to close most of this gap. Fine-tuning on labeled domain pairs helps, but even without it, testing candidates on your actual documents — not a demo dataset — surfaces domain-fit gaps before they ship.

Language Coverage: Does It Work in Every Language Your Users Write?

Language coverage is binary in a way domain fit isn't: a model either has meaningful training exposure to a language or it produces embeddings that cluster poorly for that language, degrading retrieval for every query written in it. This is a common, quiet failure in support and community products serving a global user base.

Cohere's embed-multilingual-v3.0, for instance, is documented as covering more than 100 languages in a single shared vector space — meaning a query typed in Portuguese can retrieve a passage written in English without a separate translation step. Not every candidate model offers that; some are English-only or cover a much shorter list.

If any meaningful share of your users write in a language other than English, test multilingual retrieval explicitly. Don't assume a strong English MTEB score implies acceptable Spanish, Hindi, or Japanese retrieval — it doesn't.

Dimensionality and Cost: The Trade-off Hiding in Your Infra Bill

Dimensionality and cost are the two levers with a real, calculable price tag: more dimensions per vector generally means better nuance but larger storage, slower search, and a bigger bill — and that trade-off is one a PM can and should model in a spreadsheet before committing. This is the part standard "compare model accuracy" advice tends to skip entirely.

Dimensionality: More Numbers, More Cost

Each embedding is a fixed-length vector — 384, 768, 1536, or 3072 numbers are common sizes — and every one of those numbers has to be stored and compared at query time. A larger vector isn't automatically "better"; it's a different point on a nuance-versus-cost curve.

The technique that changed this trade-off is Matryoshka Representation Learning, introduced by Kusupati and colleagues (Google Research and the University of Washington, 2022). It trains a model so the front slice of a long vector is itself a usable, coarser embedding — meaning you can truncate a 3,072-dimension vector down to 256 dimensions and still get a meaningfully useful, if less precise, embedding. OpenAI's text-embedding-3-large and text-embedding-3-small ship with this property, exposed as a dimensions parameter in their API.

Cost: The Two Bills You're Actually Paying

Cost shows up twice: once as the price of generating embeddings (an API call or compute cycle per document, usually paid once per document plus occasionally on re-embedding) and again as the ongoing cost of storing and querying the vector index. Both scale with corpus size, and only one of them is visible on a vendor's per-token pricing page.

A back-of-envelope example makes the second bill concrete: 10 million documents at 1,536 dimensions in 32-bit floats is roughly 10,000,000 × 1,536 × 4 bytes ≈ 61 GB of raw vector data, before any index overhead. Shortening to 512 dimensions cuts that to roughly 20 GB. That arithmetic belongs in the same conversation as the accuracy comparison, not a separate infra ticket discovered after launch.

A Lightweight Evaluation: Build a Labeled Query Set Instead of Trusting a Leaderboard

The reliable way to choose is a small, labeled evaluation you build yourself: 50-150 real queries against your own corpus, each with a human-judged correct answer, scored with precision@k and recall@k across your shortlisted models. This takes a few days, not a research team, and it outperforms leaderboard-chasing because it measures the thing you actually care about.

Step 1: Source Real Queries, Not Invented Ones

Pull candidate queries from support tickets, in-product search logs, or sales call transcripts — anywhere a real user already expressed a real need in their own words. Invented "sounds plausible" test queries systematically miss the phrasing, typos, and shorthand real users actually use.

This is the same discipline behind good Jobs-to-Be-Done research: you learn what people are actually trying to accomplish by listening to how they describe the job, not by guessing at a tidy paraphrase of it. A labeled query set is a JTBD interview compressed into a spreadsheet.

Step 2: Label the "Correct" Answer for Each Query

For each query, a human — ideally someone who knows the domain, like support or sales — identifies which document(s) in the corpus should have been retrieved. This is the part teams skip because it's tedious, and it's also the part that makes the whole evaluation mean anything.

• Aim for 50-150 query-answer pairs as a workable first pass; more is better, but this range already beats guessing.
• Include a mix of easy (exact-term match) and hard (paraphrased, jargon-heavy) queries — averaging only the easy ones flatters every model equally.
• Note which queries came from moments that matter most in the customer journey — a wrong retrieval during onboarding or a billing dispute costs more trust than one during idle browsing, so weight your set toward those moments.

Step 3: Score Candidates on the Same Metrics

Run each shortlisted model against the same labeled set and compare, rather than trusting a single "vibes" spot-check of a handful of outputs.

Step 4: Weigh the Score Against Domain Fit, Language Coverage, and Cost

The highest-scoring model on your labeled set isn't automatically the right choice — fold in the earlier levers before deciding.

1. Discard any candidate that fails your language-coverage requirement outright, regardless of its score.
2. Compare the top two or three finalists' scores against their dimensionality and cost trade-off, not against each other's raw accuracy alone.
3. Re-run the same labeled set any time you materially expand into a new document type or language — a model's fit isn't static as your corpus grows.
4. Budget for the fact that a model swap means re-embedding your entire corpus, so treat this decision with the weight of an infrastructure choice, not a config toggle.
5. Document which model you chose and why, alongside the evaluation numbers — not just in a Slack thread that nobody can find in six months.

Recording the Choice: Retrieval as a Packet Field

Once you've picked a model, the choice needs a durable home, not a comment in a pull request that gets buried the next time someone touches the retrieval code. This is where treating retrieval as a first-class part of your feature's spec — rather than an implementation detail — pays off.

That's the pattern worth adopting whatever tool holds it: name the model, name the bar it cleared, and write both down next to the feature they serve. It's the same discipline covered in the broader context engineering complete guide — retrieval is one of several packet-level decisions that deserve to be made on purpose.

Key Takeaways

• Retrieval quality depends on four product-level levers — domain fit, language coverage, dimensionality, and cost — not on a model's overall leaderboard rank.
• MTEB and BEIR are useful for building a shortlist of 3-4 candidates; neither predicts performance on your specific corpus, which is exactly what BEIR's own zero-shot findings show.
• Build a labeled evaluation set of 50-150 real queries, sourced from support tickets or search logs, with human-judged correct answers — this beats trusting any public benchmark.
• Score candidates on precision@k, recall@k, and MRR against that same set, then weigh the winner against language coverage and cost before deciding.
• Dimensionality is a cost lever, not just an accuracy one — Matryoshka-trained models let you truncate vectors and trade a measured amount of nuance for storage and speed.
• Swapping embedding models means re-embedding your whole corpus, so evaluate as if choosing infrastructure, not a config setting — and record the decision somewhere durable, like the retrieve field of a context packet.