Skip to main content

​
Overview

GeminiLiveVertexLLMService enables natural, real-time conversations with Google’s Gemini model through Vertex AI. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences with multimodal capabilities including audio, video, and text processing.
Want to start building? Check out our Gemini Live Guide for general concepts, then follow the Vertex AI-specific setup below.

Gemini Live Vertex API Reference

Pipecat’s API methods for Gemini Live Vertex AI integration

Example Implementation

Complete Gemini Live Vertex AI function calling example

Vertex AI Gemini Documentation

Official Vertex AI Gemini Live API documentation

Gemini Live Model Card

Gemini Live available models

​
Installation

To use Gemini Live Vertex AI services, install the required dependencies: 
pip install "pipecat-ai[google]"

​
Prerequisites

​
Google Cloud Setup

Before using Gemini Live Vertex AI services, you need:
  1. Google Cloud Project: Set up a project in the Google Cloud Console
  2. Vertex AI API: Enable the Vertex AI API in your project
  3. Service Account: Create a service account with roles/aiplatform.user and roles/ml.developer permissions
  4. Authentication: Set up service account credentials or Application Default Credentials

​
Required Environment Variables

  • GOOGLE_VERTEX_TEST_CREDENTIALS: JSON string of service account credentials (optional if using ADC)
  • GOOGLE_CLOUD_PROJECT_ID: Your Google Cloud project ID
  • GOOGLE_CLOUD_LOCATION: Vertex AI region (e.g., “us-east4”)

​
Key Features

  • Enterprise Authentication: Secure service account-based authentication
  • Multimodal Processing: Handle audio, video, and text inputs simultaneously
  • Real-time Streaming: Low-latency audio and video processing
  • Voice Activity Detection: Automatic speech detection and turn management
  • Function Calling: Advanced tool integration and API calling capabilities
  • Context Management: Intelligent conversation history and system instruction handling

​
Configuration

​
GeminiLiveVertexLLMService

This service extends GeminiLiveLLMService with Vertex AI authentication. It accepts all the same parameters as the Gemini Live service, with these differences:
​
credentials
str
default:"None"
JSON string of Google service account credentials. If not provided, falls back to credentials_path or Application Default Credentials (ADC).
​
credentials_path
str
default:"None"
Path to a service account JSON file. Used if credentials is not provided.
​
location
str
required
GCP region for the Vertex AI endpoint (e.g., "us-east4").
​
project_id
str
required
Google Cloud project ID.
​
model
str
default:"google/gemini-live-2.5-flash-native-audio"
Vertex AI model identifier to use.
​
voice_id
str
default:"Charon"
TTS voice identifier for audio responses.
​
system_instruction
str
default:"None"
System prompt for the model. Can also be provided via the LLM context.
​
tools
List[dict] | ToolsSchema
default:"None"
Tools/functions available to the model. Can also be provided via the LLM context.
​
params
InputParams
default:"InputParams()"
Runtime-configurable generation and session settings. See the Gemini Live InputParams for details.
​
start_audio_paused
bool
default:"False"
Whether to start with audio input paused.
​
start_video_paused
bool
default:"False"
Whether to start with video input paused.
​
inference_on_context_initialization
bool
default:"True"
Whether to generate a response when context is first set. Set to False to wait for user input before the model responds.
​
http_options
HttpOptions
default:"None"
HTTP options for the Google API client.

​
InputParams

The Vertex AI variant uses the same InputParams as the base Gemini Live service. See Gemini Live InputParams for the full reference.

​
Usage

​
Basic Setup with Service Account Credentials

import os
from pipecat.services.google.gemini_live import GeminiLiveVertexLLMService

llm = GeminiLiveVertexLLMService(
    credentials=os.getenv("GOOGLE_VERTEX_TEST_CREDENTIALS"),
    project_id=os.getenv("GOOGLE_CLOUD_PROJECT_ID"),
    location=os.getenv("GOOGLE_CLOUD_LOCATION"),
    voice_id="Charon",
    system_instruction="You are a helpful assistant.",
)

​
With Credentials File

llm = GeminiLiveVertexLLMService(
    credentials_path="/path/to/service-account.json",
    project_id="my-gcp-project",
    location="us-east4",
    voice_id="Puck",
    system_instruction="You are a helpful assistant.",
)

​
Using Application Default Credentials (ADC)

# When running on GCP or with gcloud auth application-default login
llm = GeminiLiveVertexLLMService(
    project_id="my-gcp-project",
    location="us-east4",
    system_instruction="You are a helpful assistant.",
)

​
With Custom Parameters

from pipecat.services.google.gemini_live import InputParams, GeminiVADParams

llm = GeminiLiveVertexLLMService(
    credentials=os.getenv("GOOGLE_VERTEX_TEST_CREDENTIALS"),
    project_id=os.getenv("GOOGLE_CLOUD_PROJECT_ID"),
    location="us-east4",
    model="google/gemini-live-2.5-flash-native-audio",
    voice_id="Charon",
    system_instruction="You are a helpful assistant.",
    params=InputParams(
        temperature=0.7,
        max_tokens=2048,
        vad=GeminiVADParams(
            silence_duration_ms=500,
        ),
    ),
)

​
Notes

  • No api_key parameter: Unlike the base GeminiLiveLLMService, Vertex AI uses service account credentials or ADC for authentication. Passing api_key will raise a ValueError.
  • Authentication priority: The service tries credentials in this order: (1) credentials JSON string, (2) credentials_path file, (3) Application Default Credentials (ADC).
  • File API not supported: The Gemini File API is not available through Vertex AI. Use Google Cloud Storage for file handling instead.
  • Model naming: Vertex AI uses different model identifiers (e.g., "google/gemini-live-2.5-flash-native-audio") compared to the Google AI variant.
  • All other features (VAD, context compression, thinking, function calling, etc.) work identically to the base Gemini Live service.