The Day I Deleted Three Pipelines

My multimodal RAG system was a Frankenstein. One embedding model for text (OpenAI text-embedding-3-large). Another for images (CLIP). Another for audio (Whisper → text → embedding). Three models, three vector indexes, “glue” code to align the spaces — and bugs at every seam.

On March 10, 2026, Google launched Gemini Embedding 2 and I deleted all three pipelines. Replaced them with a single API call returning a 3,072-dimensional vector — regardless of whether the input is text, image, video, audio, or PDF.

The feeling was like replacing three different cars (one for city, one for highway, one for off-road) with a single vehicle that does everything. Less maintenance. Fewer bugs. Fewer debugging nights. And better performance on each individual modality.

When I understood in depth how it works, I realized this model fundamentally changes how we build RAG in 2026. And connects to everything I’ve written about chunking, retrieval, and context.

The Problem It Solves (Multimodal Hell)

For those who don’t work with embeddings daily, I need to contextualize why this launch matters so much.

Embeddings are the foundation of any semantic search or RAG system. It’s the process of transforming any data — a sentence, an image, an audio clip — into a numerical vector (a list of numbers) that captures its meaning. Vectors similar in meaning cluster together in vector space. That’s how RAG “finds” relevant information.

The problem until March 2026: each data type needed a different model. Text went to text-embedding-3-large (OpenAI) or BGE-M3. Images went to CLIP or SigLIP. Audio had to be transcribed (Whisper) before becoming a text embedding. Video? No integrated solution existed.

Each model produced vectors in different spaces. A text vector and an image vector couldn’t be directly compared because they lived in distinct “mathematical universes.” For cross-modal search (“find images relevant to this sentence”), you needed alignment code, linear projections, or simply separate indexes.

The result: fragile pipelines, expensive to maintain, with performance limited by the stitching between models.

What Gemini Embedding 2 Does

Gemini Embedding 2 is Google’s first natively multimodal embedding model. A single API call accepts text, images (up to 6), video (up to 128 seconds), audio (up to 80 seconds), and PDFs (up to 6 pages) — and returns a 3,072-dimensional vector capturing the cross-modal meaning.

Available as gemini-embedding-2-preview via Gemini API and Vertex AI. Supports 100+ languages. Benchmark numbers (Google-published, awaiting full independent verification):

MTEB English: 68.32 — #1, with a 5.09-point margin over second place. The most respected embedding benchmark.

Video retrieval (Vatex, MSR-VTT, Youcook2): 68.8 — Amazon Nova 2 at 60.3 (+8.5 points). Voyage Multimodal 3.5 at 55.2 (+13.6 points).

Top-5 on MTEB Multilingual — consistent performance across 100+ languages.

Supports 8 explicit task types: semantic similarity, classification, clustering, retrieval (document and query side), code retrieval, question answering, and fact verification. Specifying the type optimizes the vector for the use case.

Matryoshka: The Russian Nesting Doll of Vectors

The technical feature that impressed me most: Matryoshka Representation Learning (MRL).

The name comes from Russian nesting dolls — each smaller doll is “nested” inside the larger one. The same principle applies to vectors: the first 768 dimensions carry the most important information. Dimensions 769-1,536 add nuance. Dimensions 1,537-3,072 add fine precision.

This means you can truncate the vector without retraining the model. Instead of using all 3,072, you can use just 768 dimensions — with minimal quality loss and 75% reduction in storage and search speed.

The math: 1 million vectors at 3,072 dimensions (float32) takes ~12 GB. At 768 dimensions: ~3 GB. For anyone indexing millions of documents, images, and videos, the difference is between “need a dedicated server” and “fits on my laptop.”

Google recommends 768 for production — “near-peak quality at one-quarter the storage cost.” Available dimensions: 128, 256, 512, 768, 1,536, 2,048, 3,072.

The Connection to Multimodal RAG

When I connect Gemini Embedding 2 with everything I’ve written about RAG, chunking, and context engineering, the impact becomes clear:

Multimodal RAG without “glue.” Before, searching “what did we discuss in the meeting about pricing?” required transcribing the audio, embedding the text, then searching. Now: embed the audio directly and search with a text query. One model. One index. One search.

Legal discovery. Everlaw already uses Gemini Embedding 2 to help lawyers search millions of records during litigation — including images and videos in case materials. Their CTO reports improved precision and recall.

Meeting knowledge bases. Embed meeting recordings directly and search with text queries. “What did we say about the pricing change?” finds the relevant segment without transcription.

Video chunking. Remember the chunking post? For text, the discussion was about tokens and semantic boundaries. For video, Gemini Embedding 2 solves differently: accepts up to 128 seconds as a single input. For longer videos, segment into 2-minute clips and embed each — creating a searchable “temporal index.”

The Price (Surprisingly Affordable)

Text: $0.20 per million tokens. Batch API: half that ($0.10/M). For comparison, OpenAI’s text-embedding-3-large costs $0.13/M but is text-only. The $0.07 difference buys 5 modalities.

Image, audio, and video follow the standard Gemini API media token rates.

For most projects, the embedding bill is a fraction of LLM inference cost. And eliminating multiple models and pipelines significantly reduces engineering cost.

The Caveats (Because Nothing Is Perfect)

Benchmarks are from Google. MTEB English is independent (and #1 is verifiable). But video benchmarks (Vatex, MSR-VTT, Youcook2) and comparisons with Amazon Nova 2 and Voyage are vendor numbers. As analyst Ewan Mak wrote: “Independent benchmarks will tell a more complete story over time. That said, the margins are large enough that even with some adjustment, the relative position likely holds.”

Input limits. 8,192 tokens for text. 6 images. 128 seconds video. 80 seconds audio. 6 PDF pages. For long documents or videos, segmentation is needed — back to the chunking problem, now across multiple modalities.

Preview, not GA. As of July 2026, still in public preview. API and pricing changes possible before GA.

Lock-in. A single Google vector space for all modalities is powerful — but creates dependency. Migrating to another provider means re-embedding everything.

Conclusion: The End of Multimodal Frankenstein

Gemini Embedding 2 marks the moment multimodal RAG stopped being a stitching exercise and became a product. One model. One vector. Five modalities. And the ability to search “the part of the meeting where we discussed the revenue chart” with a text query — no transcription, no separate pipeline, no glue.

For anyone building search, retrieval, or RAG systems in 2026: this model changes the calculus of “is multimodal search worth supporting?” from “probably not, too complex” to “why not, it’s one API call.”

And as I repeat in every technical post in this series: the technology is impressive, but real value lies in how you integrate — in the harness, the chunking, the pipeline around it. Gemini Embedding 2 eliminates an entire layer of complexity. What you do with the freed time and engineering is what will differentiate.

Share if this simplified your architecture:

Three models. Three indexes. Glue code. Bugs at every seam. Replaced with a single API call returning one vector for text, image, video, audio, and PDF. The Frankenstein is dead.

Read Also