Multimodal Embeddings Fresh 🌱

Combine multiple modalities in a single request for unified cross-modal retrieval and understanding.

Overview

Key Concept: Interleaved Input

Gemini Embedding 2 natively understands interleaved input — multiple modalities combined in a single request. This captures the complex relationships between different media types.

Single vs Multiple Entries

• Single content entry with multiple parts = one aggregated embedding
• Multiple entries in contents array = separate embeddings per entry

Step 1: Text + Image (Aggregated)

Combine text description with an image to get one unified embedding.

from google import genai
from google.genai import types

client = genai.Client()

with open('product.jpg', 'rb') as f:
    image_bytes = f.read()

# Single content with multiple parts = ONE aggregated embedding
result = client.models.embed_content(
    model='gemini-embedding-2-preview',
    contents=[
        types.Content(parts=[
            types.Part.from_text('Red leather handbag with gold hardware'),
            types.Part.from_bytes(data=image_bytes, mime_type='image/jpeg'),
        ])
    ]
)

# One unified embedding capturing both text and image
embedding = result.embeddings[0].values
print(f"Aggregated embedding: {len(embedding)} dimensions")

Step 2: Separate Embeddings for Multiple Items

Generate individual embeddings for each content entry.

from google import genai
from google.genai import types

client = genai.Client()

with open('photo1.jpg', 'rb') as f:
    img1 = f.read()
with open('photo2.jpg', 'rb') as f:
    img2 = f.read()

# Multiple content entries = SEPARATE embeddings
result = client.models.embed_content(
    model='gemini-embedding-2-preview',
    contents=[
        'A description of the first photo',
        types.Part.from_bytes(data=img1, mime_type='image/jpeg'),
        types.Part.from_bytes(data=img2, mime_type='image/jpeg'),
    ]
)

# Three separate embeddings
for i, emb in enumerate(result.embeddings):
    print(f"Entry {i+1}: {len(emb.values)} dimensions")

Step 3: Cross-Modal Search System

Build a search system where text queries find relevant images, videos, or audio.

from google import genai
from google.genai import types
import numpy as np

client = genai.Client()

def embed_text(text):
    result = client.models.embed_content(
        model='gemini-embedding-2-preview',
        contents=text,
        config=types.EmbedContentConfig(task_type='RETRIEVAL_QUERY')
    )
    return np.array(result.embeddings[0].values)

def embed_media(file_path, mime_type):
    with open(file_path, 'rb') as f:
        result = client.models.embed_content(
            model='gemini-embedding-2-preview',
            contents=[types.Part.from_bytes(data=f.read(), mime_type=mime_type)]
        )
    return np.array(result.embeddings[0].values)

def cosine_sim(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

# Index various media
media_index = [
    {'path': 'sunset.jpg', 'mime': 'image/jpeg', 'type': 'image'},
    {'path': 'podcast.mp3', 'mime': 'audio/mp3', 'type': 'audio'},
    {'path': 'demo.mp4', 'mime': 'video/mp4', 'type': 'video'},
]

# Embed all media
for item in media_index:
    item['embedding'] = embed_media(item['path'], item['mime'])

# Search with text
query_vec = embed_text('beautiful nature scenery')

results = []
for item in media_index:
    sim = cosine_sim(query_vec, item['embedding'])
    results.append((item['path'], item['type'], sim))

# Sort by similarity
results.sort(key=lambda x: x[2], reverse=True)
for path, media_type, sim in results:
    print(f"{path} ({media_type}): {sim:.4f}")

Step 4: Multimodal RAG

Combine text and image context for richer retrieval.

from google import genai
from google.genai import types

client = genai.Client()

def embed_document_with_figure(text, image_path):
    """Create a single embedding that captures both text and its figure."""
    with open(image_path, 'rb') as f:
        image_bytes = f.read()

    result = client.models.embed_content(
        model='gemini-embedding-2-preview',
        contents=[
            types.Content(parts=[
                types.Part.from_text(text),
                types.Part.from_bytes(data=image_bytes, mime_type='image/png'),
            ])
        ]
    )
    return result.embeddings[0].values

# Index documents that have both text and figures
docs = [
    {
        'text': 'Figure 1 shows the architecture of a transformer model...',
        'figure': 'transformer_arch.png'
    },
    {
        'text': 'The training loss curve in Figure 2 demonstrates convergence...',
        'figure': 'loss_curve.png'
    },
]

for doc in docs:
    doc['embedding'] = embed_document_with_figure(doc['text'], doc['figure'])

Supported Combinations

Combination	Use Case	Notes
Text + Image	Product search, visual QA	Most common multimodal pattern
Text + Audio	Podcast indexing, voice search	Captures both description and sound
Text + Video	Video search with metadata	Text provides additional context
Image + Audio	Scene understanding	Captures visual + audio context
Text + Image + Audio	Rich media indexing	Maximum context capture
Text + PDF	Annotated document search	Combines metadata with document content

Verification Checklist

• [ ] Text + Image aggregated embedding generated
• [ ] Separate embeddings for multiple entries verified
• [ ] Cross-modal search returning relevant results
• [ ] Correct understanding of single vs multiple entry behavior
• [ ] Consistent dimensions across all modality combinations
• [ ] Overall 8192 token limit not exceeded

Troubleshooting

Issue	Solution
Too many tokens	Reduce input size; total must be under 8192 tokens across all modalities
Unexpected single embedding	Check if you used Content with parts (aggregated) vs array entries (separate)
Low cross-modal scores	Use task types; ensure descriptive text matches media semantics
Mixed model embeddings	Never compare embeddings from different models

See Also

• Text Embeddings — Text-only procedures
• Image Embeddings — Image-only procedures
• RAG Pipeline — Multimodal RAG workflow
• Semantic Search — Cross-modal search architecture