Audio Embeddings Fresh 🌱

Embed audio content natively without intermediate transcription for speech search, classification, and analysis.

Overview

Specifications

Parameter	Value
Model	gemini-embedding-2-preview
Max duration	80 seconds
Supported formats	MP3, WAV
Processing	Native audio ingestion (no transcription)
Default dimensions	3,072
Pricing (Standard)	$6.50 / 1M tokens (~$0.00016 per second)
Pricing (Batch)	$3.25 / 1M tokens (~$0.00008 per second)

Step 1: Embed an Audio File (Python)

from google import genai
from google.genai import types

with open('recording.mp3', 'rb') as f:
    audio_bytes = f.read()

client = genai.Client()

result = client.models.embed_content(
    model='gemini-embedding-2-preview',
    contents=[
        types.Part.from_bytes(
            data=audio_bytes,
            mime_type='audio/mp3',
        ),
    ]
)

embedding = result.embeddings[0].values
print(f"Dimensions: {len(embedding)}")  # 3072

Step 2: Embed WAV Format

from google import genai
from google.genai import types

with open('speech.wav', 'rb') as f:
    wav_bytes = f.read()

client = genai.Client()

result = client.models.embed_content(
    model='gemini-embedding-2-preview',
    contents=[
        types.Part.from_bytes(
            data=wav_bytes,
            mime_type='audio/wav',
        ),
    ]
)

Step 3: Embed via cURL

AUDIO_BASE64=$(base64 -w 0 recording.mp3)

curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-embedding-2-preview:embedContent" \
    -H "Content-Type: application/json" \
    -H "x-goog-api-key: ${GEMINI_API_KEY}" \
    -d '{
        "content": {
            "parts": [{
                "inline_data": {
                    "mime_type": "audio/mp3",
                    "data": "'"${AUDIO_BASE64}"'"
                }
            }]
        }
    }'

Step 4: Handle Long Audio (Chunking)

For audio longer than 80 seconds, split into overlapping chunks.

from pydub import AudioSegment
from google import genai
from google.genai import types
import io

def chunk_and_embed_audio(audio_path, chunk_ms=75000, overlap_ms=5000):
    """Chunk audio into overlapping segments and embed each."""
    audio = AudioSegment.from_file(audio_path)
    client = genai.Client()
    embeddings = []
    start = 0

    while start < len(audio):
        end = min(start + chunk_ms, len(audio))
        chunk = audio[start:end]

        # Export chunk to bytes
        buffer = io.BytesIO()
        chunk.export(buffer, format='mp3')
        chunk_bytes = buffer.getvalue()

        result = client.models.embed_content(
            model='gemini-embedding-2-preview',
            contents=[types.Part.from_bytes(
                data=chunk_bytes, mime_type='audio/mp3'
            )]
        )
        embeddings.append({
            'start_ms': start,
            'end_ms': end,
            'embedding': result.embeddings[0].values
        })

        start += chunk_ms - overlap_ms

    return embeddings

Step 5: Audio Search with Text Queries

from google import genai
from google.genai import types
import numpy as np

client = genai.Client()

# Embed text query
query = client.models.embed_content(
    model='gemini-embedding-2-preview',
    contents='someone laughing',
    config=types.EmbedContentConfig(task_type='RETRIEVAL_QUERY')
)
query_vec = np.array(query.embeddings[0].values)

# Embed audio
with open('laughter.mp3', 'rb') as f:
    audio = client.models.embed_content(
        model='gemini-embedding-2-preview',
        contents=[types.Part.from_bytes(data=f.read(), mime_type='audio/mp3')]
    )
audio_vec = np.array(audio.embeddings[0].values)

similarity = np.dot(query_vec, audio_vec) / (
    np.linalg.norm(query_vec) * np.linalg.norm(audio_vec)
)
print(f"Text-to-audio similarity: {similarity:.4f}")

Native Processing

Unlike many embedding models, Gemini Embedding 2 processes audio natively — it does not transcribe audio to text first. This means it captures tonal qualities, speaker characteristics, music, and environmental sounds that transcription would miss.

Verification Checklist

• [ ] MP3 audio file embedded successfully
• [ ] WAV audio file embedded successfully
• [ ] Output vector has 3072 dimensions
• [ ] Long audio chunking implemented with overlap
• [ ] Cross-modal text-to-audio search working
• [ ] Audio under 80 seconds verified as single-request

Troubleshooting

Issue	Solution
Audio too long	Split into segments under 80 seconds with 5s overlap
Unsupported format	Convert to MP3: ffmpeg -i input.ogg output.mp3
Empty audio file	Ensure file has actual audio content, not silence
Low similarity	Audio embeddings capture more than words; use descriptive queries
Large WAV files	Convert to MP3 for smaller request size

See Also

• Video Embeddings — Embed video (includes audio track)
• Multimodal Embeddings — Combine audio with text
• Pricing — Audio embedding costs
• Semantic Search — Cross-modal search workflow