I argued against embeddings. Then I ran them on my own hardware.

Last week I wrote a post about a tool I built for answering vendor security questionnaires, and spent several paragraphs explaining why it used Postgres trigram matching instead of embeddings.

The argument went: trigram matching is deterministic, has no external dependency, adds no vendor to your own risk assessment, costs nothing per query, and is good enough for a domain with constrained vocabulary.

I’ve since replaced it with semantic search. It was a substantial improvement — the kind where you use the tool for ten minutes and can’t go back.

Here’s the thing though: every argument in that list was correct. I just had them pointed at the wrong target.

The objection was to the API, not the technique

Read my own reasoning again and notice that almost none of it is actually about embeddings. It’s about calling a hosted embedding service:

• “No external dependency” — a property of the network call, not the vector.
• “No per-token cost” — a property of the pricing model.
• “Don’t add a third-party data processor” — a property of sending your data somewhere.
• “Deterministic” — a property of a pinned model, which a versioned API endpoint is not.

Only “it’s good enough” was a claim about the matching itself, and that one turned out to be wrong.

The whole argument collapses the moment the model runs on your own machine. Which, in Elixir, it can.

Bumblebee, briefly

Bumblebee runs pre-trained Hugging Face models natively in Elixir on top of Nx, with EXLA compiling the numerical work down to native code. No Python service, no gRPC sidecar, no separate deployment.

The model is sentence-transformers/all-MiniLM-L6-v2 — a small, well-understood sentence embedding model producing 384-dimensional vectors. It’s about 90MB. It runs comfortably on the same box as the app.

def serving do
  {:ok, model_info} = Bumblebee.load_model({:hf, @model})
  {:ok, tokenizer} = Bumblebee.load_tokenizer({:hf, @model})

  Bumblebee.Text.text_embedding(model_info, tokenizer,
    output_attribute: :hidden_state,
    output_pool: :mean_pooling,
    embedding_processor: :l2_norm,
    defn_options: [compiler: EXLA]
  )
end

Mean pooling over the token hidden states, then L2 normalization so cosine similarity reduces to a dot product. That’s the standard recipe for this model family, and it’s four lines of configuration rather than four files of glue.

The part that’s genuinely Elixir’s

Here’s what made this feel different from bolting an ML model onto any other web application.

Nx.Serving goes in the supervision tree, like any other process:

{Nx.Serving,
 name: QuestionnaireCopilot.Embeddings,
 serving: QuestionnaireCopilot.Embeddings.serving(),
 batch_size: 8,
 batch_timeout: 50}

That’s the entire model-serving infrastructure. It loads the model once at boot, and it automatically batches concurrent requests — if eight embedding requests arrive within 50ms, they’re batched into one forward pass, which is dramatically more efficient than eight separate ones. Callers don’t know or care; they call Nx.Serving.batched_run/2 and get a tensor back.

In most stacks, “batch incoming inference requests with a timeout window” is a service you deploy, monitor, and page someone about. Here it’s a child spec with two tuning parameters, supervised by the same supervisor as the database pool, restarted by the same rules.

This is the payoff of the BEAM’s process model showing up somewhere I didn’t expect it. A model server is just a long-lived stateful process, and the runtime has been good at those since 1998.

pgvector, and one migration

Storage is pgvector. The migration is short enough to quote in full:

execute("CREATE EXTENSION IF NOT EXISTS vector", "DROP EXTENSION IF EXISTS vector")

alter table(:qa_pairs) do
  add :embedding, :vector, size: 384
end

create index(:qa_pairs, ["embedding vector_cosine_ops"], using: :hnsw)

An HNSW index with cosine distance. Searching is then an ORDER BY on the <=> operator:

from(q in QAPair,
  where: not is_nil(q.embedding),
  order_by: fragment("? <=> ?::vector", q.embedding, ^Pgvector.new(embedding)),
  limit: 20
)

No vector database. No second datastore to back up, secure, and keep consistent with the first one. The vectors live in the same Postgres row as the answer they describe, inside the same transaction boundary, covered by the same backup.

One detail worth stealing: the stored embedding is generated from the question concatenated with its answer, not the question alone. The answer text carries a lot of signal about what the question was really asking — a question about “data at rest” whose answer discusses AES-256 and key management becomes findable by a query about key rotation. Embedding the pair produces noticeably better retrieval than embedding the question in isolation.

Why it’s a genuine improvement

The trigram matcher was doing lexical overlap. It found “Do you encrypt data at rest?” for “Is data encrypted at rest?” because those strings share nearly every trigram.

What it could never find: “How do you protect stored customer information?” That question shares almost no character sequences with anything in the vault, and is asking exactly the same thing. Semantic search returns the encryption answer immediately.

That’s the whole category of question that made questionnaires tedious. Different security teams describe identical controls in completely different vocabulary — “at rest encryption”, “storage protection”, “data-at-rest safeguards”, “how is stored data secured”. Lexically these are four unrelated strings. Semantically they’re one question, and now the vault answers all four.

The recall improvement changed how I use the tool. With trigram matching I’d search, get nothing, and write the answer from scratch — often duplicating an answer already in the vault under different wording, which is precisely the consistency problem the vault existed to prevent.

Keeping the old path

Trigram search is still in the codebase, and still reachable. The backend is configurable, and semantic search degrades to it:

defp do_search(query, :semantic) do
  with true <- Embeddings.serving_available?(),
       embedding when is_list(embedding) <- Embeddings.generate(query) do
    semantic_search(query, embedding)
  else
    _ -> trigram_search(query)
  end
end

If the serving process isn’t running — it failed to start, the model didn’t download, someone disabled it — search silently falls back to trigram instead of returning an error. Degraded search is enormously better than no search, and this costs one with expression.

The tests take advantage of it directly:

# Don't load the ML model in tests
config :questionnaire_copilot, :start_embedding_serving, false
config :questionnaire_copilot, :search_backend, :trigram

CI doesn’t download a 90MB model or spend cycles on inference. It exercises the trigram path, which shares all the surrounding query-building and filtering logic. The one thing CI does need is the pgvector/pgvector Postgres image, since the migration creates the extension.

Backfilling existing rows is a mix task that walks every pair without an embedding and generates one. On a vault of a few hundred entries it takes under a minute.

What I’d actually take from this

Separate the technique from its delivery mechanism. My reasoning was sound and my conclusion was wrong, because I’d silently equated “embeddings” with “embedding API.” Those are different decisions with different tradeoffs, and conflating them cost me a better tool for a week. When you find yourself rejecting an approach, check whether you’re rejecting the approach or the way it’s usually packaged.

Local models are more practical than they sound. A 90MB model with a 384-dimension output, on commodity hardware, was enough to meaningfully change a real workflow. The default assumption that useful ML means an API call is increasingly just wrong.

Shipping the simple version first was still correct. This isn’t a story about having wasted a week. The trigram version got the vault populated with real answers, and a full vault is what made the embedding upgrade worth anything — swapping the matcher on an empty database would have demonstrated nothing. The simple version bought the information needed to justify the complex one. That sequence was right even though the conclusion I drew at the time wasn’t.